alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r866299432
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>>
oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient)
{
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
+ if (!isCompactionRunning) {
+ this.compactionClient.close();
+ } else {
+ // Store the old compaction client so that they can be closed
Review Comment:
Agree very strongly with the point above: unless there are very strong
argument why we can not re-init Async Service itself, i believe we should
follow t/h with an invariant that AS and Write Client lifecycles are coupled
and solve this concurrency control issues at the root
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>>
oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient)
{
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
+ if (!isCompactionRunning) {
+ this.compactionClient.close();
+ } else {
+ // Store the old compaction client so that they can be closed
Review Comment:
Immutability is hard but very powerful property we should lean in on it as
much as possible, only detour from it when there's no other choice (mostly for
perf reasons)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
