#general
@ashwanth.kumar: @ashwanth.kumar has joined the channel
@xuhongkun1103: @xuhongkun1103 has joined the channel
@karinwolok1: Hey hey! :wave: Welcome all you new Pinot slack members! :wine_glass: :heart: Would love to know who you are and what brought you here! Please take a moment and give us a short 1 liner about yourself! :ear: @kevin.dethelot @ashwanth.kumar @xuhongkun1103 @sevvy_yusuf @jvfsmrarki @weili99 @adam @fredric.rosengren @gxm.monica @polavarapu.rajaneesh @tanmeshmishra97 @mike238 @working1634 @glenn393 @kasifysf @kaashit26 @somasuntaram15 @vaibhav.goyal @madison
@adam: Thanks for the welcome and hi all. I’m an ex-dev who’s now on the sales side and this community looked like a great place to stay across Pinot!
@polavarapu.rajaneesh: 2 things. 1. Just wanted to get a few insights into running things on EKS. Especially, cloud native storage as serving file system(what would you recommend, as its mentioned
@wcxzjtz: just wondering if anyone use pyspark to read pinot data? do i just need to add the jar
#random
@ashwanth.kumar: @ashwanth.kumar has joined the channel
@xuhongkun1103: @xuhongkun1103 has joined the channel
#feat-presto-connector
@leb9882: @leb9882 has joined the channel
#feat-upsert
@leb9882: @leb9882 has joined the channel
#minion-star-tree
@leb9882: @leb9882 has joined the channel
#troubleshooting
@ashwanth.kumar: @ashwanth.kumar has joined the channel
@xuhongkun1103: @xuhongkun1103 has joined the channel
@dadelcas: How can I repair a segment external view? The segment is allocated to 2 servers out of 3 and is offline for both, the file exists in deep storage and the ideal view looks alright. I've tried reset and reload, I've also restarted the controllers and servers but the state won't change
@npawar: Any exceptions in the server logs while restarting, related to that segment?
@dadelcas: There is this error in the controller logs ```{ "id" : "1000859de3b0088__audit_event_REALTIME", "simpleFields" : { }, "mapFields" : { "HELIX_ERROR 20220307-142805.000429 STATE_TRANSITION 803412fe-7256-4e37-ac94-ef1fa200d202" : { "AdditionalInfo" : "Exception while executing a state transition task audit_event__15__19__20220305T2036Zjava.lang.reflect.InvocationTargetException\n\tat jdk.internal.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: java.lang.RuntimeException: org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.IllegalArgumentException: bound must be positive\n\tat org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadSegmentFromPeer(RealtimeTableDataManager.java:498)\n\tat org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:434)\n\tat org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:336)\n\tat org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:162)\n\tat org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:168)\n\t... 11 more\nCaused by: org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.IllegalArgumentException: bound must be positive\n\tat org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58)\n\tat org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:91)\n\tat org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadSegmentFromPeer(RealtimeTableDataManager.java:492)\n\t... 15 more\nCaused by: java.lang.IllegalArgumentException: bound must be positive\n\tat java.base/java.util.Random.nextInt(Random.java:388)\n\tat org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$1(BaseSegmentFetcher.java:92)\n\tat org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)\n\t... 17 more\n", "Class" : "class org.apache.helix.messaging.handling.HelixStateTransitionHandler", "MSG_ID" : "a9695cee-986c-4a8f-b1c8-59edfed00953", "Message state" : "READ" }, "HELIX_ERROR 20220307-142806.000618 STATE_TRANSITION 6dc67dfb-c952-4899-b46d-c4e7c2a829e6" : { "AdditionalInfo" : "Message execution failed. msgId: a9695cee-986c-4a8f-b1c8-59edfed00953, errorMsg: java.lang.reflect.InvocationTargetException", "Class" : "class org.apache.helix.messaging.handling.HelixStateTransitionHandler", "MSG_ID" : "a9695cee-986c-4a8f-b1c8-59edfed00953", "Message state" : "READ" } }, "listFields" : { } } ```
@dadelcas: Let me add more context... I was stressing the table to measure cluster resources for our use case. The server ran out heap but they kept running, is like helix swallowed the out of memory error
@dadelcas: That segment was left in a funny state where the segment url in the metadata was blank and the file had a UUID suffix in deep storage. I renamed the file by removing the UUID and updated the segment metadata in ZK. Then I called the reset segment API
@dadelcas: I did the same for other segments and they were fine, but that didn't work for this segment
@francois: Facing another issue in the way to move realtime segments to offline segment. Minion throw away ```java.lang.IllegalArgumentException: null at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.spi.creator.name.SimpleSegmentNameGenerator.generateSegmentName(SimpleSegmentNameGenerator.java:53) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:268) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:258) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.segment.processing.framework.SegmentProcessorFramework.process(SegmentProcessorFramework.java:150) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.plugin.minion.tasks.realtimetoofflinesegments.RealtimeToOfflineSegmentsTaskExecutor.convert(RealtimeToOfflineSegmentsTaskExecutor.java:164) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:135) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:58) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:111) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:88) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:831) [?:?]``` Seems to be a SegmentNameGenerator but no way to get it working :confused: ``` "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "6m", "bufferTimePeriod": "1h", "roundBucketTimePeriod": "10m", "mergeType": "concat", "maxNumRecordsPerSegment": "1000", "schedule": "* 0/10 * * * ?" } } },``` Task config. FY all my row have the same timestamp value (for testing purpose only)
@walterddr: it looks like your min/max time value contains illegal character when generating the segment name.
@walterddr: does your timestamp value contains anything like `/` ?
@francois: Timestamp is time as string :confused:
@francois: With folowing format yyyy-MM-dd’T’HH:mm:ssZ. Maybe need to pass it to timestamp using ingestionConfig
@walterddr: you can’t have `:` either. this is not allowed in some file system path.
@walterddr: see
@francois: I will try to store it as a long instead of a string to se if it makes any changes.
@weixiang.sun: I made the change to the table configuration for realtime table. But it does not take effect. Do I need reload all the segments?
@ssubrama: what change did you make?
@weixiang.sun: I added transformConfigs into ingestionConfig
@mark.needham: it will apply to the next new consuming segment AFAIK. I think it should probably work if you reload segments, as long as the messages at those offsets are still in the streaming provider.
@weixiang.sun: I tried to reload the segments. It did not work. By the way, is it possible to flush the consuming segment manually?
@mark.needham: I don't think you can. But I'm sure someone else asked about that recently. @mayanks would know
@mayanks: Transform config treated as backward incompatible change, and hence not allowed.
@mark.needham: not allowed as in it's ignored for already consumed segments?
@weixiang.sun: Transform config change is treated as backward incompatible change. It is not allowed for existing consuming segments. Is it allowed for future consuming segments?
@ssubrama: Flush segment on-demand feature is in the design process now.
@luisfernandez: hey my friends! I made a change to my zookeeper cluster, that required the entire cluster (3 nodes) to be restarted, for some reason after it got restarted for this change, (we made a change to SSD) we starting getting this ``` 2022-03-08 21:36:28,798 [myid:1] - INFO [NIOWorkerThread-1:ZooKeeperServer@1032] - Refusing session request for client /10.12.36.35:34854 as it has seen zxid 0x100000709 our last zxid is 0x0 client must try another server``` everywhere in the pinot cluster, same change was applier to our dev env but it didn’t do anything do you all know what may have caused this and how could we recover from this?
#pinot-dev
@leb9882: @leb9882 has joined the channel
@fredric.rosengren: @fredric.rosengren has joined the channel
#announcements
@leb9882: @leb9882 has joined the channel
#getting-started
@leb9882: @leb9882 has joined the channel
@fredric.rosengren: @fredric.rosengren has joined the channel
#releases
@fredric.rosengren: @fredric.rosengren has joined the channel
#minion-improvements
@leb9882: @leb9882 has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
