dlmarion commented on PR #3801: URL: https://github.com/apache/accumulo/pull/3801#issuecomment-1750784663
> > If we could leverage the non-primary managers, then this PR could likely be closed leaving the compactors the way that they are today. > > I think this PR is still, useful but maybe scoped down. Thinking of this at a really high level and seeing the following. What I am trying to puzzle out is where actual code links in to these high level concepts. Also, is the following the complete high level picture? > > Generalizing distributed work in Accumulo > > 1. Work must be found, currently TGW+TabletMgtmIterator for log sort and compaction > 2. Work must partitioned and prioritized (multiple in memory bounded prio queues that can replace stuff) > 3. Workers need to find/request work (thrift task RPCs in this PR) > 4. Work needs to be done (may need to emit status and/or metrics about work). > 5. Results of work needs to be committed (this is highly dependent on the work) There is also the issue of checking for dead workers so that the task can be queued up again. We have this today in the Manager with the DeadCompactionDetector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
