1996fanrui commented on issue #3905:
URL:
https://github.com/apache/incubator-streampark/issues/3905#issuecomment-2257842626
Thanks @HxpSerein for the detailed design, and everyone for the discussion!
I have a few of questions, please take a look in your free time, thanks!
1. In the current stage, each server includes master and worker, right?
2. From your design, the registry center has 2 implementation: zookeeper and
etcd. I'm curious could we use jdbc related database as the registry center?
Such as MySQL, PostgreSQL.
- Before this improvement, `zookeeper or etcd` is not necessary for
StreamPark. And IIUC, StreamPark needs one DataBase generally.
- If we DataBase could as the registry center, we won't introduce extra
services. And it's easy to depoly for users.
- Of course, we could provide `zookeeper or etcd` registry center in
the future if needed.
3. Why do we need to introduce the `Job Distribution` or `Consistent Hash`?
- What scenarios need Consistent Hash?
- As I understand, Consistent Hash is used in database system for
load balance.
- Database needs Consistent Hash, because it must ensure all
operations(read or write) are forwarded to the fixed server, unless the old
server is crashed.
- If key1 is written to server1, and we read it from server2. It
will be a bug(Cannot find key1 from server2).
- I don't understand why StreamPark needs Consistent Hash?
- IIUC, the current design expects each server to be responsible for
a part of jobs.
- It means, if jobA is assigned to server1, all operations of jobA
will be executed by server1.
- I don't know why all operations of jobA must be assigned to same
server1.
- For example, jobA is started by server1. After a while, user
clicks stop jobA. As I understand, server2 could stop jobA as well even if
server1 works well.
- If all operations of jobA is executed by same server isn't necessary,
I don't think we need the `Consistent Hash`. It will bring complexity to the
system.
Feel free to correct me if my understanding is wrong, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
