#general
@srsurya122: @srsurya122 has joined the channel
@srsurya122: Hello! I am very new to pinot can anyone guide me how to install and run pinot on local windows machine?
@kanth909:
@fx19880617: one thing you can try is probably running pinot in docker in windows
@srsurya122: when i try to run this command " bin/quick-start-batch.sh" by moving into pinot folder it just opens the file in notepad but it is not executing any help?
@srsurya122: thanks in advance
@fx19880617: because this `.sh` file can only run on linux
@srsurya122:
@srsurya122: When tried to execute the command(running on windows using git bash) the error is showing up all packages installed correctly $ bin/quick-start-streaming.sh Error: Could not find or load main class org.apache.pinot.tools.RealtimeQuickStart Caused by: java.lang.ClassNotFoundException: org.apache.pinot.tools.RealtimeQuickStart r
@yupeng: plz be mindful and not use at here. there are 600+ ppl in the channel. thanks
@srsurya122: Sorry my Bad wouldn't do that again.
@yupeng: no worries.
@muralidharanece: @muralidharanece has joined the channel
@charlie.gracie: @charlie.gracie has joined the channel
@mark.needham: @mark.needham has joined the channel
@ujwala.tulshigiri: Please join us for a Virtual Meetup hosted by Uber and Apache Pinot community on October 29th at 5:00PM PDT!! Please RSVP at
#random
@srsurya122: @srsurya122 has joined the channel
@muralidharanece: @muralidharanece has joined the channel
@charlie.gracie: @charlie.gracie has joined the channel
@mark.needham: @mark.needham has joined the channel
#troubleshooting
@fx19880617: @srsurya122 lets move the discussion here. What’s the environment you are running right now?
#jdbc-connector
@kharekartik: hi The issue still seem to exists
@kharekartik: after this PR This query is working ```SELECT baseballStats.playerName AS playerName, count(*) FROM baseballStats GROUP BY baseballStats.playerName ORDER BY 1 ASC```
@kharekartik: while this is not ```SELECT baseballStats.playerName AS playerName FROM baseballStats GROUP BY baseballStats.playerName ORDER BY 1 ASC```
@kharekartik: Tableau fires the second query
@kharekartik:
@fx19880617: hmm
@fx19880617: let me check
@fx19880617: any error?
@kharekartik: ``` ORDER By should be only on some/all of the columns passed as arguments to DISTINCT```
@kharekartik: ```[ { "message": "QueryValidationError:\njava.lang.UnsupportedOperationException: ORDER By should be only on some/all of the columns passed as arguments to DISTINCT\n\tat org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.validateRequest(BaseBrokerRequestHandler.java:1095)\n\tat org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:279)\n\tat org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:155)\n\tat jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)\n\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:381)\n\tat org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:819)", "errorCode": 700 } ]```
@fx19880617: I think it’s likely to be the issue of query rewrite that we rewrite the query `select a group by a` to `select distinct a`
@fx19880617: I will fix it
@kharekartik: got it but re-write will be required, right?
@kharekartik: will it work without that?
@fx19880617: I will check the order by clause should be `distinct a` or `a` I will double check
@fx19880617: it’s about order by
@fx19880617:
@fx19880617: This is the fix @kharekartik
#pinot-docs
@srsurya122: @srsurya122 has joined the channel
#metadata-push-api
@fx19880617: yes, but it will not fail my push process
@fx19880617: it is
@mayanks: interesting, how much?
@fx19880617: so I can have 100 parallelism to push segments
@fx19880617: say 100 workers
@fx19880617: each of them will prepare the segment metadata and uri then push to controller
@fx19880617: in current way, this update will fail as many worker encounters idealstates update failure
@fx19880617: with exponential backoff
@fx19880617: this change at least makes one controller update to be sequential so we can keep valid updates to idealstates
@mayanks: Is it not the same as parallelism of 3?
@mayanks: assuming 3 controllers
@mayanks: and round-robin
@fx19880617: yes from the controller side
@mayanks: then why not just do that?
@mayanks: The map increasing in case of thousands of table (corp) is a legit risk
@mayanks: you can have 100 parallelism for creating metadata
@fx19880617: hmm
@mayanks: but pushing can happening with parallelism of 3
@mayanks: we may also play with exponential backoff vs linear
@fx19880617: from spark side there is not controllable
@fx19880617: unless we may spark job 3 phase
@fx19880617: segment creation/ segment metadta creation / segment metadata push
@mayanks: first phase can also do metadata?
@fx19880617: you mean apart from segment tar file we also write a metadata file beside it ?
@mayanks: yes
@mayanks: it is already there, we just need to copy before tarring
@mayanks: that can be an option to segment generation
@fx19880617: hmm, then basically we are creating two tar files and save to deepstore
@mayanks: phase 2 is doing that explicitly?
@fx19880617: one is metadata with data file and the other is metadata only
@mayanks: and doing it as second pass
@fx19880617: yes
@mayanks: yeah, so i think we don't need 2 passes for that
@fx19880617: it needs to download from s3 and untar it
@fx19880617: then tar metadata file again
@mayanks: that is inefficient
@mayanks: what if create-segment had option to create two files, one tar, and one metadata
@mayanks: then you don't need to download from s3, untar....
@fx19880617: we still need to
@fx19880617: just download metadata tar file from s3
@fx19880617: it’s smaller though
@mayanks: yeah but you save on downloading the segment tar to compute metadata
@mayanks: that will be a big saving
@fx19880617: true, but this can be solved by giving many workers
@fx19880617: but the controller IS update is not
@mayanks: I feel we are trying to fix the symptom of too many retries and failure
@mayanks: I agree that is a bad symptom
@mayanks: But I think there may be a better fix to fix the root cause
@mayanks: If we can understand why we are running into retries and failures
@fx19880617: one thing I was thinking here is can we make the IS update to be in batch mode
@fx19880617: say we update multiple segments together
@mayanks: It is counter intuitive if you think it like this: ```One one side, we are increasing paralleism to 100 On the other side, we are reducing it to 3```
@fx19880617: the reason is obvious
@mayanks: Yeah, I like batch update idea
@fx19880617: each thread is reading the old IS, then apply the change then try to update
@fx19880617: however IS is already updated
@mayanks: I think there is either a design doc or a PR, that is making the push a transaction
@fx19880617: right now it is transaction
@fx19880617: with the IS version
@mayanks: Oh ok
@mayanks: No, I mean from the spark side
@fx19880617: that’s why the fail happens and retry
@steotia:
@mayanks: So extending your idea @fx19880617 :
@mayanks: Say each controller gathers all segments in local IS (without updating)
@mayanks: then at the end of transaction of push, updates the IS
@fx19880617: that IS update will fail, then retry will read current IS then apply the change of adding one segment agai n
@mayanks: We need this for versioning
@mayanks: transaction here is from client side: ```push start -> push data -> push end```
@mayanks: needs more thinking as each controller needs to handle push-end. However, in general I like the idea
@fx19880617: ``` HelixHelper.updateIdealState(_helixZkManager, offlineTableName, idealState -> { assert idealState != null; Map<String, Map<String, String>> currentAssignment = idealState.getRecord().getMapFields(); if (currentAssignment.containsKey(segmentName)) { LOGGER.warn("Segment: {} already exists in the IdealState for table: {}, do not update", segmentName, offlineTableName); } else { List<String> assignedInstances = segmentAssignment.assignSegment(segmentName, currentAssignment, instancePartitionsMap);
@mayanks: of reducing updates
@fx19880617: this is one update per segment
@fx19880617: if the given IS is not current, then it will fail
@fx19880617: then retry
@mayanks: Yeah, so bulk update will reduce retries, right
@fx19880617: yes
@mayanks: Can we think along those lines. I think it will be useful for versioning of data as well
@mayanks: And also atomic swap of IS. We have customers who want to reduce data inconsistency during push, so want to see either all old or all new segments at once
@fx19880617: sync will make the update to be sequential but no version conflict
@fx19880617: IS swap is atomic
@fx19880617: the update will provide current IS version
@fx19880617: if that doesn’t match then IS update will fail
@mayanks: I am referring to refresh use case, where all segments should be swapped atomically
@mayanks: we can have a quick zoom call?
@mayanks: might be faster/easier to converge
@fx19880617: after 1pm ?
@mayanks: yes
@fx19880617: I have meeting from 10:30 to 1pm
@mayanks: works
@fx19880617: k
@steotia: I will also join. Need some context on the above
@fx19880617: cool
@mayanks: i can fill you in sid
@steotia: awesome
@steotia: will digest the slack convo in the meantime
@fx19880617: can you send the invitation ?
@mayanks: what's your email id
@fx19880617:
@mayanks: sent invite for 1:30pm zoom link:
@fx19880617: cool
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
