pgondi13 commented on issue #9289:
URL: https://github.com/apache/hudi/issues/9289#issuecomment-2524708197
I set `.option("hoodie.datasource.read.file.index.listing.mode", "eager")`
(instead of the default `"lazy"`) when reading from hudi, and that seemed to
have fixed the exception. This behavior change happened in
https://github.com/apache/hudi/pull/6680. My theory is that since we are
running multiple threads that are accessing different partitions of the hudi
table, the `cachedAllInputFileSlices` map that holds all the file information
for each partition is being concurrently modified. I'm not sure how to avoid
this while still having lazy evaluation, but hopefully using eager will
continue to be supported and is the cause of our issues. For our use case, we
do computation on each partition, so lazy evaluation does not help us much
since we will eventually need all the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
