I am thinking of implementing the load based rebalancing based on the google algorithm "Consistent Hashing with Bounded Loads," I am glad Helix team is working on a weighted rebalancing algorithm as well, that would help me a lot. Any idea when this feature can be released?
On Sun, Oct 20, 2019 at 11:44 PM Hunter Lee <[email protected]> wrote: > As Kishore said, Helix powers Apache Pinot. > > Apache Gobblin also uses Helix for cluster management and Helix Task > Framework (online workflow scheduler) for its data ingestion sync use > cases, running 80K+ jobs per day. > > Hunter > > On Sun, Oct 20, 2019 at 10:06 PM kishore g <[email protected]> wrote: > >> Can you talk about your use case?. Helix does not rebalance partitions >> based on load automatically but one can use customized mode to achieve >> that. >> >> On Sun, Oct 20, 2019 at 10:01 PM Lei Xia <[email protected]> wrote: >> >>> Here are blogs with some latest adaptation of Helix outside of LinkedIn. >>> >>> >>> - Uber's Kafka replicator built on top of Helix: >>> https://eng.uber.com/ureplicator/ >>> >>> - Pinterest's RocksDB replicator: >>> >>> https://medium.com/@Pinterest_Engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>> >>> - >>> - Airbnb’s Change Data Capture system: >>> >>> https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f >>> >>> >>> >>> >>> *Lei Xia* >>> >>> >>> Data Infra/Helix >>> >>> [email protected] >>> www.linkedin.com/in/lxia1 >>> ------------------------------ >>> *From:* kishore g <[email protected]> >>> *Sent:* Sunday, October 20, 2019 9:24 PM >>> *To:* [email protected] <[email protected]> >>> *Subject:* Re: who uses helix >>> >>> At LinkedIn, Helix manages thousands on nodes. 1 million segments >>> (equivalent of partitions) across thousands of tables. >>> >>> On Sun, Oct 20, 2019 at 9:16 PM Jianzhou Zhao <[email protected]> wrote: >>> >>> Cool. >>> >>> In the rocksplicator case, how many partitions, replica + nodes a helix >>> cluster can manage? >>> >>> On Sun, Oct 20, 2019 at 9:05 PM Bo Liu <[email protected]> wrote: >>> >>> We extensively use Helix at Pinterest. This blog post has more details >>> and some tips. >>> >>> https://medium.com/pinterest-engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Fpinterest-engineering%2Fautomated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=If0JOZJSZFBq2jziFG5ZGdhoE%2B30KHw%2B4a2V6F56fLo%3D&reserved=0> >>> >>> On Sun, Oct 20, 2019 at 8:30 PM Jianzhou Zhao <[email protected]> wrote: >>> >>> Hi, >>> >>> I was looking for who uses helix, and got >>> https://cwiki.apache.org/confluence/display/HELIX/Powered+By+Helix >>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHELIX%2FPowered%2BBy%2BHelix&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=5HtQ%2F8U3IvrrLjhoQN4%2BlO9r9jNDpuX9MeXVGovKKAs%3D&reserved=0> >>> >>> The link did not get updated after 2017. Is the list still update to >>> date? >>> >>> Thank you, j >>> >>> >>> >>> -- >>> Best regards, >>> Bo >>> >>>
