<h3><u>#general</u></h3><br><strong>@sharmavedang07: </strong>@sharmavedang07
has joined the channel<br><strong>@bobli.usc: </strong>@bobli.usc has joined
the channel<br><strong>@karinwolok1: </strong>@karinwolok1 has joined the
channel<br><strong>@sriyansh.cse: </strong>@sriyansh.cse has joined the
channel<br><h3><u>#random</u></h3><br><strong>@sharmavedang07:
</strong>@sharmavedang07 has joined the channel<br><strong>@bobli.usc:
</strong>@bobli.usc has joined the channel<br><strong>@karinwolok1:
</strong>@karinwolok1 has joined the channel<br><strong>@sriyansh.cse:
</strong>@sriyansh.cse has joined the
channel<br><h3><u>#sql-rollout-plan</u></h3><br><strong>@tim780:
</strong>@tim780 has joined the
channel<br><h3><u>#minion-star-tree</u></h3><br><strong>@mailtobuchi:
</strong>okay. thanks. I just got this idea and wanted to explore. Seems like
it’s possible. We may or may not implement immediately but will definitely ping
here if we’re proceeding with impl. appreciate the help<br><strong>@laxman:
</strong>@laxman has joined the
channel<br><h3><u>#troubleshooting</u></h3><br><strong>@sjeetsingh2801:
</strong>@sjeetsingh2801 has joined the channel<br><strong>@yash.agarwal:
</strong>Hey Team, I am getting intermittent exceptions in CombinePlanNode.
```Exception processing requestId 137
java.lang.RuntimeException: Caught exception while running CombinePlanNode.
at
org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:149)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:221)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:155)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:139)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_265]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_265]
at
shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_265]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
~[?:1.8.0_265]
at
org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:139)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-701ffcbd5be5f39e91cea9a0297c4e8b0a7d9343]
... 13 more
Processed
requestId=137,table=guestslslitm3years_OFFLINE,segments(queried/processed/matched/consuming)=1058/-1/-1/-1,schedulerWaitMs=0,reqDeserMs=4,totalExecMs=10659,resSerMs=0,totalTimeMs=10663,minConsumingFreshnessMs=-1,broker=Broker_10.59.100.47_8099,numDocsScanned=-1,scanInFilter=-1,scanPostFilter=-1,sched=fcfs
Slow query: request handler processing time: 10663, send response latency: 58,
total time to handle request: 10721```
Is there a reason why this is happening ? Is there a way we can override the
timeout of 10s in CombineNodePlan<br><strong>@yash.agarwal: </strong>What is
the best approach to solve the same ? I am currently storing about 3 Billion
rows (1000 segments) per table on a single data node. Should I rebalance it to
more servers or add CPU/RAM to the same.<br><strong>@tim780: </strong>@tim780
has joined the channel<br><strong>@tim780: </strong>hey
<#C011C9JHN7R|troubleshooting> i have followed the getting started guide for
running pinot in kubernetes. i was able to configure a table to ingest records
from a kafka cluster.<br><strong>@tim780: </strong>i was able to ingest 125000
records<br><strong>@tim780: </strong>when i do a count(*) on the table the
count never increases<br><strong>@tim780: </strong>i see the following in the
controller log<br><strong>@tim780: </strong>```2020/09/01 22:22:31.308 WARN
[LLCSegmentCompletionHandlers] [grizzly-http-server-1] Segment file:
file:/var/pinot/controller/data/motion/motion__3__0__20200901T2214Z already
exists. Replacing it with segment: motion__3__0__20200901T2214Z from instance:
Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
2020/09/01 22:22:31.311 WARN [LLCSegmentCompletionHandlers]
[grizzly-http-server-0] Segment file:
file:/var/pinot/controller/data/motion/motion__6__0__20200901T2214Z already
exists. Replacing it with segment: motion__6__0__20200901T2214Z from instance:
Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
2020/09/01 22:22:31.320 ERROR
[SegmentCompletionFSM_motion__3__0__20200901T2214Z] [grizzly-http-server-1]
Caught exception while committing segment metadata for segment:
motion__3__0__20200901T2214Z
java.lang.NullPointerException: null
at
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.updateCommittingSegmentZKMetadata(PinotLLCRealtimeSegmentManager.java:507)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:446)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:416)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1091)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:656)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentCommitEnd(SegmentCompletionManager.java:325)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommit(LLCSegmentCompletionHandlers.java:330)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0^Cfa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:446)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:416)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1091)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
at
org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:656)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]```<br><strong>@tim780:
</strong>why does it say segment file already exists?<br><strong>@tim780:
</strong><br><strong>@tim780: </strong>the cluster manager ui says status good
or consuming<br><strong>@jackie.jxt: </strong>Hi @tim780, based on the
exception, I think the time column is not configured
correctly<br><strong>@jackie.jxt: </strong>Can you please share the table
config and the schema?<br><strong>@tim780: </strong>```{
"schemaName":"motion",
"dimensionFieldSpecs":[
{
"name":"sensor_id",
"dataType":"STRING"
},
{
"name":"config_details",
"dataType":"STRING"
},
{
"name":"has_motion",
"dataType":"BOOLEAN"
}
],
"metricFieldSpecs":[
],
"dateTimeFieldSpecs":[
{
"name":"time_stamp",
"dataType":"LONG",
"format":"1:MILLISECONDS:EPOCH",
"granularity":"1:MILLISECONDS"
}
]
}```<br><strong>@tim780: </strong>```{
"tableName":"motion",
"tableType":"REALTIME",
"segmentsConfig":{
"timeColumnName":"timestampInEpoch",
"timeType":"MILLISECONDS",
"schemaName":"motion",
"replicasPerPartition":"1"
},
"tenants":{
},
"tableIndexConfig":{
"loadMode":"MMAP",
"streamConfigs":{
"streamType":"kafka",
"stream.kafka.consumer.type":"lowlevel",
"stream.kafka.topic.name":"ash-logger.etldb.derived.motion",
"stream.kafka.decoder.class.name":"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list":"kafka-dev-1-kafka-bootstrap.kafka-dev-1:9092",
"realtime.segment.flush.threshold.time":"3600000",
"realtime.segment.flush.threshold.size":"50000",
"stream.kafka.consumer.prop.auto.offset.reset":"smallest"
}
},
"metadata":{
"customConfigs":{
}
}
}```<br><strong>@tim780: </strong>an example value for `time_stamp` is
`1599000470000`<br><strong>@npawar: </strong>table config says
```"segmentsConfig":{
"timeColumnName":"timestampInEpoch",```<br><strong>@tim780: </strong>ooh
i think i screwed up in `timeColumnName` for config<br><strong>@tim780:
</strong>i feel dumb<br><strong>@tim780: </strong>i just copy pasted this
config from the example<br><strong>@tim780: </strong>sorry to bother; i will
correct and retry and report back<br><strong>@npawar: </strong>no worries! lmk
how it goes<br><strong>@tim780: </strong>quick question. i used
`pinot-admin.sh` to create this table. how would i go about dropping it and
recreating it with the proper config?<br><strong>@npawar: </strong>you can drop
using swagger APIs. if you go to localhost:9000 you should see this
option<br><strong>@npawar: </strong>assuming 9000 is your controller
port<br><strong>@tim780: </strong>i will try; thank you<br><strong>@tim780:
</strong>thank you @npawar and @jackie.jxt<br><strong>@tim780: </strong>it
works now<br><strong>@tim780: </strong>i do see some `2020/09/01 23:09:14.532
WARN [TopStateHandoffReportStage]
[HelixController-pipeline-default-pinot-(d1b54062_DEFAULT)] Event
d1b54062_DEFAULT : Cannot confirm top state missing start time. Use the current
system time as the start time.`<br><strong>@tim780: </strong>in the controller
logs<br><strong>@tim780: </strong>but at least i see count(*)
updating<br><strong>@jackie.jxt: </strong>@tim780 This warning is from Helix,
which you can ignore<br><strong>@tim780: </strong><br><strong>@tim780:
</strong>i don’t know what to make of the segments marked
bad<br><strong>@pradeepgv42: </strong>Hi, fyi with the latest master, swagger
endpoint seems to be broken. Not sure if anyone
noticed<br><h3><u>#pinot-k8s-operator</u></h3><br><strong>@tim780:
</strong>@tim780 has joined the channel<br>
