#general
@noahprince8: Is it possible to pull data for one table from multiple kafka topics with the same message type over them. Doesn't look like you can do a subscribePattern
@g.kishore: No, we don't have that ability right now. This has been requested a few times now
@mayanks: Perhaps file an issue, if there isn't one already?
@noahprince8: Will do.
@sjeetsingh2801: Hello Team, can someone please provide me write access to
@steotia: Hi Sarabjeet, you should create a pull request. Are you seeing issues when creating PR?
@mayanks: Hello, write access is open for committers. Is there a PR that is already approved?
@sjeetsingh2801: Nope, I’m unable to push my changes to the repo. Getting a 403 forbidden..
@mayanks: You need to create a fork and push changes there. And then file PR from there
@sjeetsingh2801: ahhhh I see. My bad! This is my first time contributing. I apologize. I’ll do it and create a PR soon. Thanks :slightly_smiling_face:
@mayanks: No worries, and thanks for contributing
@femosobakin: @femosobakin has joined the channel
@pabraham.usa: @pabraham.usa has joined the channel
@pabraham.usa: Hi All, I am new to Pinot and got some basic questions. When comparing with ElasticSearch do Pinot creates similar large scale index? Can I configure Pinot to talk to S3 or other data stores directly without provisioning additional space for local Indices? if in case local index is mandatory how big it will be? Thanks.
@mayanks: At the moment, local index is needed. There's a PR in progress for using S3/deepstore as primary storage, and caching in data locally as needed.
@mayanks:
@pabraham.usa: Thanks @mayanks, Great to know that a PR is going on . Meanwhile is there a possibility to restrict the local index to store say 12 hours of data and rotate it? and fall back to S3 for other queries even though there will be a latency?
@chinmay.cerebro: Pinot does support data retention. However for query federation, you'll have to do additional work today. One way to do it is using Presto + Pinot on realtime data (0-12 hours) and Presto + S3 for everything else. Of course, the federation is not quite seamless (i.e. same query cannot span across both today).
@pabraham.usa: @chinmay.cerebro This looks like a possible soln. Do we have to expose two different urls/ui for this or somehow use superset to provide single interface to the user?
@chinmay.cerebro: With either Presto or superset, you don't need 2 different URLs. However internally they'll be treated as different catalogs or databases
@chinmay.cerebro: so the queries have to explicitly mention that. I'm not sure if there's a good way to hide it
@pabraham.usa: Thanks @chinmay.cerebro , will start looking in this direction. I assume I can leverage superset custom viz to do that. Not sure I can do the same with presto.
@chinmay.cerebro: With Presto, all you do is add 2 different catalogs
@chinmay.cerebro: One for Pinot: (see this:
@chinmay.cerebro: and the other one for S3
@pabraham.usa: Thank you, I anyways require presto for S3 integration so using it is definitely a good approach. Do Pinot have any helm charts?
@chinmay.cerebro: yes
@chinmay.cerebro:
@pabraham.usa: Perfect..! seems I got everything to crack in.
@chinmay.cerebro: awesome ! do let us know how it goes. This is not new btw, at Uber we ran Presto + Pinot & HDFS in a federated manner
@chinmay.cerebro: that was insanely useful since data scientists could query both pinot and Hive data in the same query
@pabraham.usa: will do , also the chart seems to have presto and superset as well. Thats is a good thinking from whoever designed it.
@chinmay.cerebro: I think it was @fx19880617 :slightly_smiling_face:
#random
@femosobakin: @femosobakin has joined the channel
@pabraham.usa: @pabraham.usa has joined the channel
#troubleshooting
@karinwolok1: @karinwolok1 has joined the channel
@joe.quinn: @joe.quinn has joined the channel
@noahprince8: ```Cloning into '/opt/pinot-build'... fatal: unable to access '
@fx19880617: seems missing ca file for https
@fx19880617: ```
#presto-pinot-connector
@noahprince8: @noahprince8 has joined the channel
@noahprince8:
@g.kishore: No
@noahprince8: Ah. Might end up adding that down the line. Could be really useful with that time based segment pruner.
@g.kishore: yes
@g.kishore: at some point, we should just merge pinot broker and presto worker
@fx19880617: Agreed. There are Pinot segment page and Pinot broker page. Pinot broker page will directly query Pinot broker, so Pinot broker will handle that logic. For Pinot segment page, presto will query every Pinot segment. It’s great to have the pruning feature in presto as well.
@fx19880617: Typical segment query has predicate, so time based pruning will be very useful
@noahprince8: Where would I create an issue for this? In incubator-pinot or on the presto github?
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
