alexeykudinkin commented on code in PR #7642:
URL: https://github.com/apache/hudi/pull/7642#discussion_r1087292737
##########
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java:
##########
@@ -73,11 +73,12 @@ public abstract class BaseTableMetadata implements
HoodieTableMetadata {
private static final Logger LOG =
LogManager.getLogger(BaseTableMetadata.class);
- public static final long MAX_MEMORY_SIZE_IN_BYTES = 1024 * 1024 * 1024;
- public static final int BUFFER_SIZE = 10 * 1024 * 1024;
+ protected static final long MAX_MEMORY_SIZE_IN_BYTES = 1024 * 1024 * 1024;
+ // NOTE: Buffer-size is deliberately set pretty low, since MT internally is
relying
+ // on HFile (serving as persisted binary key-value mapping) to do
caching
+ protected static final int BUFFER_SIZE = 10 * 1024; // 10Kb
Review Comment:
Create a ticket to make HFile block configurable
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/data/HoodieJavaRDD.java:
##########
@@ -119,7 +118,16 @@ public <O> HoodieData<O>
mapPartitions(SerializableFunction<Iterator<T>, Iterato
@Override
public <O> HoodieData<O> flatMap(SerializableFunction<T, Iterator<O>> func) {
- return HoodieJavaRDD.of(rddData.flatMap(e -> func.apply(e)));
+ // NOTE: Unrolling this lambda into a method reference results in
[[ClassCastException]]
+ // due to weird interop b/w Scala and Java
+ return HoodieJavaRDD.of(rddData.flatMap(x -> func.apply(x)));
+ }
Review Comment:
Oh, i recollect the context now -- i've just added a comment here b/c it was
failing after i've accidentally converted it to method reference, so left it to
caution future readers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
