Exact timeline may vary, but it's currently in active development and I'd say sometime in the first half of 2020.
Hunter On Mon, Oct 21, 2019 at 8:39 AM Yi Chen <[email protected]> wrote: > I am thinking of implementing the load based rebalancing based on the > google algorithm "Consistent Hashing with Bounded Loads," I am glad Helix > team is working on a weighted rebalancing algorithm as well, that would > help me a lot. Any idea when this feature can be released? > > On Sun, Oct 20, 2019 at 11:44 PM Hunter Lee <[email protected]> wrote: > >> As Kishore said, Helix powers Apache Pinot. >> >> Apache Gobblin also uses Helix for cluster management and Helix Task >> Framework (online workflow scheduler) for its data ingestion sync use >> cases, running 80K+ jobs per day. >> >> Hunter >> >> On Sun, Oct 20, 2019 at 10:06 PM kishore g <[email protected]> wrote: >> >>> Can you talk about your use case?. Helix does not rebalance partitions >>> based on load automatically but one can use customized mode to achieve >>> that. >>> >>> On Sun, Oct 20, 2019 at 10:01 PM Lei Xia <[email protected]> wrote: >>> >>>> Here are blogs with some latest adaptation of Helix outside of LinkedIn. >>>> >>>> >>>> - Uber's Kafka replicator built on top of Helix: >>>> https://eng.uber.com/ureplicator/ >>>> >>>> - Pinterest's RocksDB replicator: >>>> >>>> https://medium.com/@Pinterest_Engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>>> >>>> - >>>> - Airbnb’s Change Data Capture system: >>>> >>>> https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f >>>> >>>> >>>> >>>> >>>> *Lei Xia* >>>> >>>> >>>> Data Infra/Helix >>>> >>>> [email protected] >>>> www.linkedin.com/in/lxia1 >>>> ------------------------------ >>>> *From:* kishore g <[email protected]> >>>> *Sent:* Sunday, October 20, 2019 9:24 PM >>>> *To:* [email protected] <[email protected]> >>>> *Subject:* Re: who uses helix >>>> >>>> At LinkedIn, Helix manages thousands on nodes. 1 million segments >>>> (equivalent of partitions) across thousands of tables. >>>> >>>> On Sun, Oct 20, 2019 at 9:16 PM Jianzhou Zhao <[email protected]> wrote: >>>> >>>> Cool. >>>> >>>> In the rocksplicator case, how many partitions, replica + nodes a helix >>>> cluster can manage? >>>> >>>> On Sun, Oct 20, 2019 at 9:05 PM Bo Liu <[email protected]> wrote: >>>> >>>> We extensively use Helix at Pinterest. This blog post has more details >>>> and some tips. >>>> >>>> https://medium.com/pinterest-engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Fpinterest-engineering%2Fautomated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=If0JOZJSZFBq2jziFG5ZGdhoE%2B30KHw%2B4a2V6F56fLo%3D&reserved=0> >>>> >>>> On Sun, Oct 20, 2019 at 8:30 PM Jianzhou Zhao <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> I was looking for who uses helix, and got >>>> https://cwiki.apache.org/confluence/display/HELIX/Powered+By+Helix >>>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHELIX%2FPowered%2BBy%2BHelix&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=5HtQ%2F8U3IvrrLjhoQN4%2BlO9r9jNDpuX9MeXVGovKKAs%3D&reserved=0> >>>> >>>> The link did not get updated after 2017. Is the list still update to >>>> date? >>>> >>>> Thank you, j >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Bo >>>> >>>>
