abhijeetkushe opened a new issue #8070:
URL: https://github.com/apache/pinot/issues/8070
Table Config
```{
"tableName": "workflowEvents",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "eventTimestamp",
"timeType": "MILLISECONDS",
"schemaName": "workflowEvents",
"replicasPerPartition": "1",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "365",
"segmentPushType": "APPEND"
},
"tenants": {
"broker":"DefaultTenant",
"server":"DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kinesis",
"stream.kinesis.topic.name": "prod-rel-cdp-dl-workflow-metrics-stream",
"region": "us-east-1",
"shardIteratorType": "LATEST",
"stream.kinesis.consumer.type": "lowlevel",
"stream.kinesis.fetch.timeout.millis": "30000",
"stream.kinesis.decoder.class.name":
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kinesis.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
"realtime.segment.flush.threshold.size": "5000000",
"realtime.segment.flush.threshold.time": "1d"
}
},
"upsertConfig": {
"mode": "FULL"
},
"routing": {
"instanceSelectorType": "strictReplicaGroup"
},
"metadata": {
"customConfigs": {}
}
}
```
- I upgraded pinot to 0.9.1 version from 0.8.0 last month and after that I
started seeing GetRecord.IteratorAgeInMilliseconds metrics displaying in
correct values
- Yesterday I observed the data stopped being displayed in the Table from
the Query Console even though the GetRecords metrics showed that records were
read but where not read from the Table.There were some new data fields that
were added to the record but the schema was not updated
- I decided to change Table config to AT_SEQUENCE_NUMBER and I did not see
the changes immediately but this morning I was able to see those records but
post that when records were added this morning they were not being reflected in
the table
- I decided to switch the Table config back to LATEST.I also saw that the
old segments got terminated and new segments were created but the records were
still not reflected in the Table
So both the data issue and GetRecord.IteratorAgeInMilliseconds still persist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
