As Kishore said, Helix powers Apache Pinot. Apache Gobblin also uses Helix for cluster management and Helix Task Framework (online workflow scheduler) for its data ingestion sync use cases, running 80K+ jobs per day.
Hunter On Sun, Oct 20, 2019 at 10:06 PM kishore g <[email protected]> wrote: > Can you talk about your use case?. Helix does not rebalance partitions > based on load automatically but one can use customized mode to achieve > that. > > On Sun, Oct 20, 2019 at 10:01 PM Lei Xia <[email protected]> wrote: > >> Here are blogs with some latest adaptation of Helix outside of LinkedIn. >> >> >> - Uber's Kafka replicator built on top of Helix: >> https://eng.uber.com/ureplicator/ >> >> - Pinterest's RocksDB replicator: >> >> https://medium.com/@Pinterest_Engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >> >> - >> - Airbnb’s Change Data Capture system: >> >> https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f >> >> >> >> >> *Lei Xia* >> >> >> Data Infra/Helix >> >> [email protected] >> www.linkedin.com/in/lxia1 >> ------------------------------ >> *From:* kishore g <[email protected]> >> *Sent:* Sunday, October 20, 2019 9:24 PM >> *To:* [email protected] <[email protected]> >> *Subject:* Re: who uses helix >> >> At LinkedIn, Helix manages thousands on nodes. 1 million segments >> (equivalent of partitions) across thousands of tables. >> >> On Sun, Oct 20, 2019 at 9:16 PM Jianzhou Zhao <[email protected]> wrote: >> >> Cool. >> >> In the rocksplicator case, how many partitions, replica + nodes a helix >> cluster can manage? >> >> On Sun, Oct 20, 2019 at 9:05 PM Bo Liu <[email protected]> wrote: >> >> We extensively use Helix at Pinterest. This blog post has more details >> and some tips. >> >> https://medium.com/pinterest-engineering/automated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833 >> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Fpinterest-engineering%2Fautomated-cluster-management-and-recovery-for-rocksplicator-f1f8fd35c833&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=If0JOZJSZFBq2jziFG5ZGdhoE%2B30KHw%2B4a2V6F56fLo%3D&reserved=0> >> >> On Sun, Oct 20, 2019 at 8:30 PM Jianzhou Zhao <[email protected]> wrote: >> >> Hi, >> >> I was looking for who uses helix, and got >> https://cwiki.apache.org/confluence/display/HELIX/Powered+By+Helix >> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHELIX%2FPowered%2BBy%2BHelix&data=02%7C01%7Clxia%40linkedin.com%7Cbf9310fdb70647d3761308d755df905f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637072291002932821&sdata=5HtQ%2F8U3IvrrLjhoQN4%2BlO9r9jNDpuX9MeXVGovKKAs%3D&reserved=0> >> >> The link did not get updated after 2017. Is the list still update to date? >> >> Thank you, j >> >> >> >> -- >> Best regards, >> Bo >> >>
