estebanz01 opened a new issue, #12547:
URL: https://github.com/apache/pinot/issues/12547
Hola! 👋
I'm having some trouble with a pinot cluster deployed into kubernetes with
minion enabled. I want to move data from real time table to offline table but
it's failing with the following information:
```java
^T[16:18:00.383 [TaskStateModelFactory-task_thread-7] ERROR
org.apache.pinot.minion.taskfactory.TaskFactoryRegistry - Caught exception
while executing task:
Task_RealtimeToOfflineSegmentsTask_8961b037-3c41-47d7-b56f-375ef16dc2fc_1709569080105_0
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed
after 1 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:62)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:158)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:152)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:202)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:190)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:201)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:77)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
That's the only error I see in the minion pods and there's nothing else on
the other pods for pinot. Any ideas on how to debug this further? here's the
schema and table config for my hybrid table:
<details>
<summary>Schema definition</summary>
```json
{
"schemaName": "data_counting",
"dimensionFieldSpecs": [
{
"name": "device_name",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "bytes_sent",
"dataType": "LONG"
}
],
"dateTimeFieldSpecs": [
{
"name": "__key",
"dataType": "TIMESTAMP",
"format": "1:MICROSECONDS:EPOCH",
"granularity": "1:MICROSECONDS"
},
{
"name": "__metadata$eventTime",
"dataType": "TIMESTAMP",
"format": "1:MICROSECONDS:EPOCH",
"granularity": "1:MICROSECONDS"
}
]
}
```
</details>
<details>
<summary>Table configuration (REALTIME)</summary>
```json
{
"REALTIME": {
"tableName": "data_counting_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"schemaName": "data_counting",
"replication": "1",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "15",
"replicasPerPartition": "1",
"minimizeDataMovement": false,
"timeColumnName": "__key"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant",
"tagOverrideConfig": {}
},
"tableIndexConfig": {
"invertedIndexColumns": [],
"noDictionaryColumns": [],
"streamConfigs": {
"streamType": "pulsar",
"stream.pulsar.topic.name": "persistent://client/devices/all",
"stream.pulsar.bootstrap.servers":
"pulsar://pulsar-proxy.pulsar.svc.cluster.local:6650",
"stream.pulsar.prop.auto.offset.reset": "smallest",
"stream.pulsar.consumer.type": "lowlevel",
"stream.pulsar.fetch.timeout.millis": "20000",
"stream.pulsar.decoder.class.name":
"org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
"stream.pulsar.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory",
"realtime.segment.flush.threshold.rows": "10000",
"realtime.segment.flush.threshold.time": "1h",
"stream.pulsar.metada.populate": "true",
"stream.pulsar.metadata.fields": "eventTime"
},
"loadMode": "MMAP",
"onHeapDictionaryColumns": [],
"varLengthDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"rangeIndexColumns": [],
"rangeIndexVersion": 2,
"optimizeDictionary": false,
"optimizeDictionaryForMetrics": false,
"noDictionarySizeRatioThreshold": 0.85,
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"sortedColumn": [],
"bloomFilterColumns": []
},
"metadata": {},
"quota": {},
"task": {
"taskTypeConfigsMap": {
"RealtimeToOfflineSegmentsTask": {
"bucketTimePeriod": "1h",
"bufferTimePeriod": "2h",
"mergeType": "concat",
"maxNumRecordsPerSegment": "100000",
"schedule": "0 * * * * ?"
}
}
},
"routing": {},
"query": {
"timeoutMs": 60000
},
"ingestionConfig": {
"continueOnError": false,
"rowTimeValueCheck": false,
"segmentTimeValueCheck": true
},
"isDimTable": false
}
}
```
</details>
<details>
<summary>Table configuration (OFFLINE)></summary>
```json
{
"OFFLINE": {
"tableName": "data_counting_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"schemaName": "data_counting",
"replication": "1",
"replicasPerPartition": "1",
"timeColumnName": "__key",
"minimizeDataMovement": false,
"segmentPushType": "APPEND",
"segmentPushFrequency": "HOURLY"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"invertedIndexColumns": [],
"noDictionaryColumns": [],
"rangeIndexColumns": [],
"rangeIndexVersion": 2,
"createInvertedIndexDuringSegmentGeneration": false,
"autoGeneratedInvertedIndex": false,
"sortedColumn": [],
"bloomFilterColumns": [],
"loadMode": "MMAP",
"onHeapDictionaryColumns": [],
"varLengthDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"optimizeDictionary": false,
"optimizeDictionaryForMetrics": false,
"noDictionarySizeRatioThreshold": 0.85
},
"metadata": {},
"quota": {},
"routing": {},
"query": {},
"ingestionConfig": {
"continueOnError": false,
"rowTimeValueCheck": false,
"segmentTimeValueCheck": true
},
"isDimTable": false
}
}
```
</details>
I'm using apache pulsar 3.2.0 and apache pinot version 1.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
