mayankshriv opened a new issue #6029: URL: https://github.com/apache/incubator-pinot/issues/6029
The current code simply assumes partitionId from kafka partition to be in accordance with the partition function defined in table config. In case there is a mis-match on kafka partition and what's set in table config, that leads to incorrect pruning of segments causing bad results for query. There was indeed an old behavior were we wiped out the incorrect partition metadata when committing the segment, which would limit the scope to consuming queries. At the very minimum, we should bring back that behavior. This is important as we can run into these issues in various situations: * Increase in number of partitions from Kafka side. * Partition function change on either side (accidental or intentional). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
