Let’s start another thread to discuss the specific feature and the timeline.
On Mon, Oct 21, 2019 at 2:21 PM Hunter Lee <[email protected]> wrote: > Exact timeline may vary, but it's currently in active development and I'd > say sometime in the first half of 2020. > > Hunter > > On Mon, Oct 21, 2019 at 8:39 AM Yi Chen <[email protected]> wrote: > >> I am thinking of implementing the load based rebalancing based on the >> google algorithm "Consistent Hashing with Bounded Loads," I am glad Helix >> team is working on a weighted rebalancing algorithm as well, that would >> help me a lot. Any idea when this feature can be released? >> >> On Sun, Oct 20, 2019 at 11:44 PM Hunter Lee <[email protected]> wrote: >> >>> As Kishore said, Helix powers Apache Pinot. >>> >>> Apache Gobblin also uses Helix for cluster management and Helix Task >>> Framework (online workflow scheduler) for its data ingestion sync use >>> cases, running 80K+ jobs per day. >>> >>> Hunter >>> >>> On Sun, Oct 20, 2019 at 10:06 PM kishore g <[email protected]> wrote: >>> >>>> Can you talk about your use case?. Helix does not rebalance partitions >>>> based on load automatically but one can use customized mode to achieve >>>> that. >>>> >>>> On Sun, Oct 20, 2019 at 10:01 PM Lei Xia <[email protected]> wrote: >>>> >>>>> Here are blogs with some latest adaptation of Helix outside of >>>>> LinkedIn. >>>>> >>>>> >>>>> - Uber's Kafka replicator built on top of Helix: >>>>> https://eng.uber.com/ureplicator/ >>>>> >>>>> - Pinterest's RocksDB replicator: >>>>> >>>>> https://medium.com/@Pinterest_Engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>>>> >>>>> - >>>>> - Airbnb’s Change Data Capture system: >>>>> >>>>> https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f >>>>> >>>>> >>>>> >>>>> >>>>> *Lei Xia* >>>>> >>>>> >>>>> Data Infra/Helix >>>>> >>>>> [email protected] >>>>> www.linkedin.com/in/lxia1 >>>>> ------------------------------ >>>>> *From:* kishore g <[email protected]> >>>>> *Sent:* Sunday, October 20, 2019 9:24 PM >>>>> *To:* [email protected] <[email protected]> >>>>> *Subject:* Re: who uses helix >>>>> >>>>> At LinkedIn, Helix manages thousands on nodes. 1 million segments >>>>> (equivalent of partitions) across thousands of tables. >>>>> >>>>> On Sun, Oct 20, 2019 at 9:16 PM Jianzhou Zhao <[email protected]> wrote: >>>>> >>>>> Cool. >>>>> >>>>> In the rocksplicator case, how many partitions, replica + nodes a >>>>> helix cluster can manage? >>>>> >>>>> On Sun, Oct 20, 2019 at 9:05 PM Bo Liu <[email protected]> wrote: >>>>> >>>>> We extensively use Helix at Pinterest. This blog post has more details >>>>> and some tips. >>>>> >>>>> https://medium.com/pinterest-engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >>>>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Fpinterest-engineering%2Fautomated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=If0JOZJSZFBq2jziFG5ZGdhoE%2B30KHw%2B4a2V6F56fLo%3D&reserved=0> >>>>> >>>>> On Sun, Oct 20, 2019 at 8:30 PM Jianzhou Zhao <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I was looking for who uses helix, and got >>>>> https://cwiki.apache.org/confluence/display/HELIX/Powered+By+Helix >>>>> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHELIX%2FPowered%2BBy%2BHelix&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=5HtQ%2F8U3IvrrLjhoQN4%2BlO9r9jNDpuX9MeXVGovKKAs%3D&reserved=0> >>>>> >>>>> The link did not get updated after 2017. Is the list still update to >>>>> date? >>>>> >>>>> Thank you, j >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Bo >>>>> >>>>>
