Jackie-Jiang opened a new pull request, #9095:
URL: https://github.com/apache/pinot/pull/9095
Make the following enhancement to the upsert metadata manager:
- Add replace segment support
- Log error and emit metric (`PARTIAL_UPSERT_ROWS_NOT_REPLACED`) for
segment not fully replaced, which can potentially cause inconsistency between
replicas for partial upsert table
- Remove the remaining primary keys from the replaced segment immediately
so that new consuming segment is not affected
- Handle empty segment properly
- Enhance the log to log the table name, partition id and primary key count
- Clean up the code and move the upsert related logic into the metadata
manager, such as creating the record info iterator
- In `IndexSegment`, replace `getPrimaryKey()` with `getValue()` which is
more general and can be used to read the comparison column as well
- Fix the issue of using BYTES as primary key
## Release Notes
Added a metric `PARTIAL_UPSERT_ROWS_NOT_REPLACED` to track the rows not
replaced for partial upsert table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
