keith-turner opened a new issue, #3517:
URL: https://github.com/apache/accumulo/issues/3517
Currently in Accumulo user compactions for a table always store
configuration at the same place in ZK. User compactions also have a per tablet
compaction counter that is used by the manager to know when all compactions are
done for a user initiated table compaction. The per table zookeeper storage and
per tablet counter are not great for concurrent compactions. Accumulo
currently does not allow concurrent compactions on a table with different
configuration.
#3513 implemented user compactions in the manager but it intentionally did
not fully implement the existing model described above. Rather than carry
forward the above limited model, it would be better to implement a new model
that supports multiple concurrent compactions with different configuration.
This new model would have two components.
1. Storing per user compaction config in ZK under a node that is unique to
that compaction. This node could use the fate tx id for its name. This node
would replace the current per table node that stores compaction config.
2. Adding a new column to the tablet of the form family=compacted and
qualifier=<fate transaction id>. This column indicates whether or not an
individual fate operation has compacted a tablet. It would replace the current
compact count. The compaction count is impossible to reason about w/
concurrent compactions.
With the above information multiple user compactions could run concurrently
on a table and even overlap. The following example scenario is not currently
possible in Accumulo should be able to run under this new model.
1. A table has three tablet : tab1, tab2, and tab3
2. user compaction1 that covers tab1 and tab2 is started with configA
(iterators that age off certain data).
3. user compaction2 that covers tab2 and tab3 is started with configB
(iterators that filter certain data).
4. tablets tab1 and tab2 start compacting with configA for user compaction 1
5. tablet tab3 starts compacting with config B for user compaction 2, but
can not start a compaction on tab2 because one is running.
6. The compactions for tab1 and tab2 complete and they are marked as
complete using the new compacted column. The fate tx id to mark completion
comes from the new new selected column introduced in #3513. The compaction
coordinator marks it complete on compaction commit.
7. The fate operation for user compaction 1 sees all tablets are complete
and reports back to the user.
8. The fate operation for user compaction 2 sees that tab2 can now be
compacted and starts a new compaction using config B.
9. The compactions for tab2 and tab3 as part of user compaction 2 complete
and go through the same process as described above.
In the example above both user compactions overlapped a single tablet tab2.
That tablet was compacted twice, first with configA and then second with
configB. The order could have been reversed, the configB compaction could
have gone first followed by the configA compaction. Under this new system when
concurrent user compactions overlap the overlapping tablets will be compacted
once for each user compaction in some arbitrary order.
Under this new system compaction cancellation would need to change.
Currently it uses the per table compaction counter. Under the new system it
would need to find all user compactions that are currently running and mark
each one as canceled in ZK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
