shounakmk219 opened a new pull request, #10905:
URL: https://github.com/apache/pinot/pull/10905
Currently if you add a new StarTree index to the table we end up throwing
all existing StarTree indexes and rebuild all from scratch, not the new one
only. Same goes for a StarTree index deletion, it will rebuild the rest of the
trees again.
This in a scenario where data size is large and there are bunch of existing
indexes can take multiple days to rebuild.
This PR tries to enhance the process by providing a way to reuse the pre
built startrees.
**Logical flow**
```
1. Separate and extract the existing startrees
1.1. Separate out the combined index file into separate index files
1.2. Extract the index map file
2. Delete the startrees
3. Iterate over the new startrees list
3.1. If startree is present in the extracted existing startrees
3.1.1. Copy the pre existing index files
3.1.2. Update the metadata properties
3.2. Else
3.2.1. Build the startree from the configs
3.3. Combine the reused/generated index files
4. Write the files to segment directory
5. Cleanup created temp files.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
