Rachelint opened a new issue, #1480:
URL: https://github.com/apache/incubator-horaedb/issues/1480
### Describe This Problem
We found in production that the speed of sst compaction is unable to keep up
with the speed of sst generation, leading to poor query performance... However
we are unable give more resource to compaction to solve the problem because
query/write is more important than compaction in the same node.
It is really hard to do a trade-off about resource allocation among query,
write and compaction in lsm model... We want to compact the generated small
ssts as fast as possible, but we can't tolerate its influence to query/write.
And finally I think offload the compaction to the seperated nodes may be the
key for it.
### Proposal
For supporting compaction offload, we need:
- Special node supporting remote compaction service
- [ ] Impl compaction service
- Horaedb node supports submitting the real compaction node to remote
- [ ] Refactor the compaction process and define necessary traits
- [ ] Impl remote mode compactor based on traits above
- Horaemeta supports managing the special compaction nodes
- [ ] Impl the ability to manage the compaction nodes
- [ ] Expose the api for horaedb node to get the proper remote compaction
ndoe
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
