rahil-c commented on code in PR #18341:
URL: https://github.com/apache/hudi/pull/18341#discussion_r2966728284
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -109,8 +118,55 @@ public HoodieSparkLanceWriter(StoragePath file,
public HoodieSparkLanceWriter(StoragePath file,
StructType sparkSchema,
TaskContextSupplier taskContextSupplier,
- HoodieStorage storage) {
- this(file, sparkSchema, null, taskContextSupplier, storage, false,
Option.empty());
+ HoodieStorage storage,
+ long maxFileSize) {
+ this(file, sparkSchema, null, taskContextSupplier, storage, false,
Option.empty(), maxFileSize);
+ }
+
+ /**
+ * Constructor for Spark Lance writer used for internal row writing with
pre-embedded metadata
+ * and a configurable file size limit.
+ *
+ * @param file Path where Lance file will be written
+ * @param sparkSchema Spark schema for the data
+ * @param instantTime Instant time for the commit
+ * @param taskContextSupplier Task context supplier for partition ID
+ * @param storage HoodieStorage instance
+ * @param populateMetaFields Whether to populate Hudi metadata fields
+ * @param bloomFilterOpt Optional bloom filter for record key tracking
+ * @throws IOException if writer initialization fails
+ */
+ public HoodieSparkLanceWriter(StoragePath file,
+ StructType sparkSchema,
+ String instantTime,
+ TaskContextSupplier taskContextSupplier,
+ HoodieStorage storage,
+ boolean populateMetaFields,
+ Option<BloomFilter> bloomFilterOpt) {
+ this(file, sparkSchema, instantTime, taskContextSupplier, storage,
populateMetaFields, bloomFilterOpt, Long.MAX_VALUE);
Review Comment:
Shouldn't we have some reasonable default here for a `maxFileSize` rather
than `Long.MAX` Value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
