keith-turner opened a new pull request, #5247:
URL: https://github.com/apache/accumulo/pull/5247
This change fixes #5188. Unfortunately it touches a lot of code because of
cascading dependencies in the code. It would be difficult to break this into a
smaller commit. These changes do reduce some of those dependencies though.
There are two major advantages after this change. First tablet metadata is
no longer kept in memory for queued compactions. Second the tablet metadata is
no longer read during compaction reservation. Before this change the following
would happen.
1. TGW would find a tablet to compact and enqueue compaction jobs+tablet
metadata.
2. Eventually when a compactor requested a job it would yank job+tablet
metadata off the queue.
3. To reserve the compaction a lot of complex analysis was done in the
coordinator and then a conditional mutation was submitted. The conditional
mutation would require all data involved in the complex analysis to be the same.
4. If the conditional mutation failed then the coordinator would reread the
tablet metadata and go back to step 3.
After this change the following happens in the code.
1. TGW would find a tablet to compact and enqueue a new class called
ResolvedCompactionJob. This new class takes the compaction job and tablet
metadata and computes all information needed for the compaction later. The
TabletMetadata object is no longer refrenced by this class after the
constructor returns. So this class will use much less memory on the queue for
the case when tablet have lots of files.
2. Eventually when a compactor requested a job it would yank a
ResolvedCompactionJob off the queue.
3. All of the complex analysis to determine if a compaction can start is
now done in the conditional mutation instead of in the coordinator. To enable
this, new functionality was added to Ample including the new
TabletMetadataCheck interface, TabletMetadataCheckIterator, and the
CompactionReservationCheck class. With these changes its now super easy to
write a conditional check that will do arbitrary analysis of TabletMetadata
prior to committing a mutation.
4. Since the analysis is done in the conditional mutation there is no
longer a need to retry. If the mutation fails then we know the compaction can
not run.
The following were some supporting changes that had to be made.
* Took methods for encoding KeyExtent as base64 from
TabletManagementParameters and moved the KeyExtent because this was needed in
the new TabletMetadataCheckIterator.
* Move TabletMetadata out of the compaction queues, which made those more
independent but was a big change. The main change here is that instead of
adding `TabletMetadata, List<CompactionJob>` to the compaction queue now
`KeyExtent, List<CompactionJob>` is added. This required changing the test for
these classes and the code that interacts with them in CompactionCoordinator
creating lots of diff.
* Moved CompactionCoordinatorTest.testCanReserve() to
CompactionReservationCheckTest.testCanReserve() and changed the code to work
with the new CompactionReservationCheck class. These are test of the complex
logic that used to run in the coordinator and now runs in the tablet server as
part of a conditional mutation.
* Removed conditional checks from Ample related to compactions that were no
longer used after these changes. There was code for requiring a set of files
not not be compacting.
The new TabletMetadataCheck functionality of ample could be used to simplify
other conditional mutation checks of tablet metadata. It made the compaction
reservation code much simpler and easier to understand. This code could be
further improved if #5244 were implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
