haibow opened a new pull request #4954: Support schema evolution for consuming segments URL: https://github.com/apache/incubator-pinot/pull/4954 Implement the design in https://github.com/apache/incubator-pinot/issues/4225#issuecomment-548173120 Support querying newly added columns in the consuming segment (OFF by default to keep default behavior the same). Currently after updating the schema for a realtime table and calling reload on the table, consuming segments would not be reloaded with the new schema, which would cause consuming segments to be dropped at query time due to schema inconsistency. In this diff, we use the virtual column provider to provide default null values for the newly added columns in the consuming segment (until it is committed), to address the schema inconsistency issue. There is still a delay in indexing and querying the actual values in the newly added columns (up to a few hours, depending on the segment commit thresholds). We will revisit this later. This functionality is off by default for now to keep the default behavior the same.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
