rohityadav1993 opened a new pull request, #13837:
URL: https://github.com/apache/pinot/pull/13837
`feature` `release-notes`
Follow up of #13107
This PR adds support for uploading batch generated segments to an upsert
table. A new segmentName convention `UploadedRealtimeSegmentName` was
introduced which enables segment assignment for uploaded segments(not generated
by stream consumers) based on the provided partition number.
Changes:
- Adds new config in BatchConfigMap needed for
`UploadedRealtimeSegmentNameGenerator`
- Adds a new integration test for upsert tables
- Refactor README with an example to quick start.
Test plan:
- Integration test
- Cluster test:
- Set up a new upsert table with keys partitioned using murmur2
- Generate segments from a Flink job and upload to table
- Validation:
- Check the keys of a uploaded segment are consistently partitioned with
realtime stream generated segments
- <img width="760" alt="Screenshot 2024-08-19 at 2 26 49 PM"
src="https://github.com/user-attachments/assets/56466699-cfc9-4155-b0e1-07ac04fcc02e";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
