#general
@xwu: @xwu has joined the channel
@mark.needham: I wrote a quick blog post showing how to get rid of 'bad instances' from the Pinot UI. I often get int this state after restarting my machine and then starting up local clusters again -
@arthurvanzyl: @arthurvanzyl has joined the channel
@kishorenaidu712: Hi, I was performing stream ingestion through kafka in a standalone machine. I had 5 partitions created and hence 5 segments in pinot. The parameter "segment.flush.threshold.size" is set to 10000. When i try ingesting data with 100k records, only 50k records are available. Will the flushing of consuming segment take time to update or is 50k the upper bound for the configuration mentioned ?
@mark.needham: do you mean you only see 50k records in pinot?
@mayanks: Size is in bytes, if you want to control rows there is s different setting
@kishorenaidu712: Yes, i am able to view only 50k records
@mark.needham: hmmm. The flushing doesn't affect whether you can view them - the flushing is only when does a new segment get created. But you should be able to see the records as soon as they are ingested from kafka
@kishorenaidu712: What actually happens if a segment reaches max limit of any set parameter maybe rows,size or time?
@mark.needham: the segment will be committed to the deep store and a new one will be created. The new one will be where any new messages are ingested
@kishorenaidu712: Okay, then why am I not able to view the entire records?
@mark.needham: you can check which kafka offsets Pinot has processed by partition
@mayanks: Use the table debug endpoint to check if an issues, I am wondering if the consumption stopped
@mark.needham: the debug endpoint is here -
@kishorenaidu712: Sure, will check for it. Thank you
@mayanks: You can also check server log for any errors
@ltejaswi8: @ltejaswi8 has joined the channel
@rsivakumar: @rsivakumar has joined the channel
@yo: @yo has joined the channel
#random
@xwu: @xwu has joined the channel
@arthurvanzyl: @arthurvanzyl has joined the channel
@ltejaswi8: @ltejaswi8 has joined the channel
@rsivakumar: @rsivakumar has joined the channel
@yo: @yo has joined the channel
#feat-compound-types
@arthurvanzyl: @arthurvanzyl has joined the channel
#feat-text-search
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#feat-rt-seg-complete
@arthurvanzyl: @arthurvanzyl has joined the channel
#feat-presto-connector
@arthurvanzyl: @arthurvanzyl has joined the channel
#feat-upsert
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-helix
@arthurvanzyl: @arthurvanzyl has joined the channel
#group-by-refactor
@arthurvanzyl: @arthurvanzyl has joined the channel
#order-by
@arthurvanzyl: @arthurvanzyl has joined the channel
#feat-better-schema-evolution
@arthurvanzyl: @arthurvanzyl has joined the channel
#fraud
@arthurvanzyl: @arthurvanzyl has joined the channel
#inconsistent-segment
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-power-bi
@arthurvanzyl: @arthurvanzyl has joined the channel
#minion-star-tree
@arthurvanzyl: @arthurvanzyl has joined the channel
#troubleshooting
@shaileshjha061: Hi Team I am Getting the below error for JMX Exporter. (Server, Broker, Controller) Pods Not able to start. Kindly assist on the same. Is there any change for enabling the JMX Exporter?? ```Error opening zip file or JAR manifest missing : /opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar Error occurred during initialization of VM agent library failed to init: instrument``` Thanks Team @dlavoie
@navina: @shaileshjha061 there was a change to use 0.16 jar (
@shaileshjha061: Thanks @navina for replying back. I am using the helm chart to install the Pinot Cluster in K8s and configuring to expose metric following this doc:
@navina: In this step, can you try changing `0.12.0` to `0.16.1` and see if it helps? I think the docker image now packages `0.16.1` . I haven't tried the tutorial myself. but maybe try my suggestion.
@shaileshjha061: Sure Let me try this out. Will update here Thanks
@navina: @shaileshjha061 I tried it with `/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar` and it brings up the pinot cluster without errors. if the version change doesn't work, you can try this too.
@shaileshjha061: Great Thanks
@shaileshjha061: will try this too.
@dlavoie: @shaileshjha061 Can you force your update of the docker image? A bug was merged on the prometheus path this weekend and was fixed yesterday
@dlavoie: FYI:
@xwu: @xwu has joined the channel
@elon.azoulay: Hi, we noticed that the download url for all segments in our upsert tables is null, but the segments exist in deepstore. Has anyone ever seen that issue before? Thinking to manually update all the segment metadata in zk since they are realtime segments. lmk if that is how you resolve this.
@mayanks: Hmm, this seems quite odd. Personally, I haven’t seen this. Are there any errors in the logs during segment commit?
@elon.azoulay: I will search for it, since we're using peer download should we look in the server or the controller?
@elon.azoulay: also, we rebalanced the upsert tables after scaling up a tenant, not sure if that had something to do with it.
@elon.azoulay: thanks for responding @mayanks! I will search the logs and see what I can find
@elon.azoulay: Seeing messages like this: ``` 2022/02/20 02:28:50.884 WARN [PinotFSSegmentUploader] [enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z] Failed to upload file /var/pinot/server/data/index/enriched_station_orders_v1_16_2_upsert_REALTIME/enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z.tar.gz of segment enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z for table java.util.concurrent.TimeoutException ```
@elon.azoulay: But it seems to have uploaded the files
@elon.azoulay: I will download a few just to check them
@elon.azoulay: It seems that all the upsert table segments have that error, but we have realtime tables that do not have the error
@elon.azoulay: Is there a way to force pinot to retry the upload the segments from the servers on the upsert tenant?
@elon.azoulay: I see the timeout is hardcoded, I can try to increase it - but same cluster has servers which upload all segments within 10 seconds: ```PinotFSSegmentUploader.DEFAULT_SEGMENT_UPLOAD_TIMEOUT_MILLIS```
@mayanks: @yupeng any ideas ^^
@elon.azoulay: I extrapolated the segment upload time from some of our metrics - maybe gcs is slow for us, it's only 225mb per segment so it shouldn't take 10 seconds :man-shrugging:
@elon.azoulay: I actually see the file there on gcs
@yupeng: if it times out and fails to upload, then no url is expected?
@mayanks: But @yupeng Elon mentioned that the segment was indeed pushed to deepstore (or at least exists there).
@yupeng: well, it's possible that the file uploaded but fail to receive response from client side
@yupeng: also, i think the segment upload flow is the same for non-upsert tables, so i dont see anything special with upsert
@elon.azoulay: Thanks! I'm trying with an increased timeout
@arthurvanzyl: @arthurvanzyl has joined the channel
@ayush.jha: Hi everyone a quick question if the `pinot.broker.timeoutMs` is greater than `pinot.server.query.executor.timeout` what will be the actual timeout of the query it will be of the server or of the broker??
@richard892: in between
@richard892: at `pinot.server.query.executor.timeout` the servers will all start timing out, but the broker will wait for them all to timeout unless that happens after `pinot.broker.timeoutMs`
@richard892: that's probably a complicated way of saying set them both to the same value
@ayush.jha: Okayy thanks understood
@mayanks: Yes set this value to the latency value your app can’t tolerate.
@apte.kaivalya: Hey :wave: , I am trying the `INCREMENT` upsert config type using something like ```"upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "countOfEvents": "INCREMENT", "type": "OVERWRITE" }, "defaultPartialUpsertStrategy": "OVERWRITE", "hashFunction": "MURMUR3" },``` On upserts I see that the `type` field was overwritten however `countOfEvents` didn’t `increment` . Am I missing something?
@apte.kaivalya: Hmm maybe I misunderstood? the doc says: ```INCREMENT Add the new value to the existing values``` this field isn’t coming from Kafka so it will be no value, defaultNullValue is set 1 so I’d expect it will add 1 to the value.
@ltejaswi8: @ltejaswi8 has joined the channel
@rsivakumar: @rsivakumar has joined the channel
@yo: @yo has joined the channel
#pinot-s3
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-k8s-operator
@arthurvanzyl: @arthurvanzyl has joined the channel
#onboarding
@arthurvanzyl: @arthurvanzyl has joined the channel
#feat-geo-spatial-index
@arthurvanzyl: @arthurvanzyl has joined the channel
#custom-aggregators
@arthurvanzyl: @arthurvanzyl has joined the channel
#docs
@arthurvanzyl: @arthurvanzyl has joined the channel
#aggregators
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-dev
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#community
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#announcements
@arthurvanzyl: @arthurvanzyl has joined the channel
@mosiac: @mosiac has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#s3-multiple-buckets
@arthurvanzyl: @arthurvanzyl has joined the channel
#multiple_streams
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#lp-pinot-poc
@arthurvanzyl: @arthurvanzyl has joined the channel
#presto-pinot-connector
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#multi-region-setup
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#pinot-realtime-table-rebalance
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#time-based-segment-pruner
@arthurvanzyl: @arthurvanzyl has joined the channel
#discuss-validation
@arthurvanzyl: @arthurvanzyl has joined the channel
#segment-cold-storage
@arthurvanzyl: @arthurvanzyl has joined the channel
#config-tuner
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-perf-tuning
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#thirdeye-pinot
@arthurvanzyl: @arthurvanzyl has joined the channel
#getting-started
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#feat-partial-upsert
@arthurvanzyl: @arthurvanzyl has joined the channel
#releases
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#debug_upsert
@arthurvanzyl: @arthurvanzyl has joined the channel
#flink-pinot-connector
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-rack-awareness
@arthurvanzyl: @arthurvanzyl has joined the channel
#minion-improvements
@arthurvanzyl: @arthurvanzyl has joined the channel
#complex-type-support
@arthurvanzyl: @arthurvanzyl has joined the channel
#pinot-docsrus
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
#pinot-trino
@arthurvanzyl: @arthurvanzyl has joined the channel
#udf-type-matching
@arthurvanzyl: @arthurvanzyl has joined the channel
@rsivakumar: @rsivakumar has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
