dongkelun commented on a change in pull request #4016:
URL: https://github.com/apache/hudi/pull/4016#discussion_r752789562
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
##########
@@ -339,6 +339,11 @@ public boolean isBeforeTimelineStarts(String instant) {
return details.apply(instant);
}
+ @Override
+ public boolean isEmpty(HoodieInstant instant) {
+ return getInstantDetails(instant).get().length == 0;
Review comment:
Hello, I'm not sure why the file size is 0, because I don't fully grasp
the source code level. But I think it may be because the concurrency is too
high (writing multiple tables at the same time) or the program exits abnormally
due to network reasons, or it may be related to the following exception:
```scala
Caused by: java.lang.RuntimeException:
org.apache.hudi.exception.HoodieException:
org.apache.hudi.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
hoodie-hdfs:/warehouse/tablespace/managed/hive/hudi_nync.db/td_cbdkxx/fda63f47-10ad-4353-85c6-801080823843-0_0-0-0_20210923103444.parquet
does not have a open stream. Cannot get the bytes written on the stream
at
org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.lambda$execute$0(BaseJavaCommitActionExecutor.java:120)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:116)
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:66)
at
org.apache.hudi.table.action.commit.AbstractWriteHelper.write(AbstractWriteHelper.java:55)
... 17 more
ed by: org.apache.hudi.exception.HoodieException:
org.apache.hudi.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
hoodie-hdfs:/warehouse/tablespace/managed/hive/hudi_nync.db/td_cbdkxx/fda63f47-10ad-4353-85c6-801080823843-0_0-0-0_20210923103444.parquet
does not have a open stream. Cannot get the bytes written on the stream
at
org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:73)
at
org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:38)
at
org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 23 more
ed by: org.apache.hudi.exception.HoodieException:
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
hoodie-hdfs:/warehouse/tablespace/managed/hive/hudi_nync.db/td_cbdkxx/fda63f47-10ad-4353-85c6-801080823843-0_0-0-0_20210923103444.parquet
does not have a open stream. Cannot get the bytes written on the stream
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:147)
at
org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:69)
... 25 more
ed by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException:
hoodie-hdfs:/warehouse/tablespace/managed/hive/hudi_nync.db/td_cbdkxx/fda63f47-10ad-4353-85c6-801080823843-0_0-0-0_20210923103444.parquet
does not have a open stream. Cannot get the bytes written on the stream
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 26 more
Caused by: java.lang.IllegalArgumentException:
hoodie-hdfs:/warehouse/tablespace/managed/hive/hudi_nync.db/td_cbdkxx/fda63f47-10ad-4353-85c6-801080823843-0_0-0-0_20210923103444.parquet
does not have a open stream. Cannot get the bytes written on the stream
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.getBytesWritten(HoodieWrapperFileSystem.java:985)
at
org.apache.hudi.io.storage.HoodieParquetWriter.canWrite(HoodieParquetWriter.java:90)
at
org.apache.hudi.io.HoodieCreateHandle.canWrite(HoodieCreateHandle.java:122)
at
org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:88)
at
org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
```
In addition, to explain the scenario where the file size is 0, we integrate
Java Hudi into nifi, and then many tables write data with Java Hudi at the same
time. This exception may occur. Colleagues are also accompanied by other
exceptions, but other exceptions have rollback or will not be thrown
continuously, which will not cause data loss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
