1996fanrui commented on issue #3905: URL: https://github.com/apache/incubator-streampark/issues/3905#issuecomment-2258198801
>> 2.From your design, the registry center has 2 implementations: zookeeper and etcd. I'm curious could we use jdbc related database as the registry center? Such as MySQL, PostgreSQL. > > The specific implementation plan is still under discussion. In my opinion, both solutions have their own advantages and disadvantages. Using a single database can simplify system architecture and management, while using ZooKeeper or etcd has distributed consistency and high availability. In general, database has distributed consistency and high availability as well. As I understand, StreamPark relies heavily on the database. All user information and job information are stored in the database, so the StreamPark cannot work after database is crashed even if the registry center is using zookeeper. If we introduced the zookeeper, StreamPark will be unavailable whenever either Zookeeper or database crashes. This increases the maintenance cost for users and makes StreamPark more likely to be unavailable. > > 3.Why do we need to introduce the Job Distribution or Consistent Hash? > The server needs to poll jobs to obtain their latest status. When the number of jobs is huge, monitoring can become a significant challenge. Therefore, in my opinion, consistent hashing is primarily used to ensure load balancing in monitoring. In cases of cluster expansion or disaster, the consistent hashing algorithm can help migrate job monitoring. As my example mentioned before, all operations of same key should be forwarded to the same server for database system. But I still don't understand why same job should be monitored in the same server? Or my question is: if job is monitored by one random server, does it works? If yes, there is no need to introduce complex consistent hashing. I may have missed some background information, sorry for the question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
