aokolnychyi commented on a change in pull request #3292:
URL: https://github.com/apache/iceberg/pull/3292#discussion_r742537954
##########
File path:
spark/v3.0/spark/src/main/java/org/apache/iceberg/spark/actions/Spark3BinPackStrategy.java
##########
@@ -61,9 +62,17 @@ public Table table() {
SparkSession cloneSession = spark.cloneSession();
cloneSession.conf().set(SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
false);
+ long targetReadSize = splitSize(inputFileSize(filesToRewrite));
+ // Ideally this would be the row-group size but the row group size is
not guaranteed to be consistent
+ long fileSplitSize = Long.valueOf(table.properties().getOrDefault(
+ TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES,
Review comment:
It looks a bit odd to simply check the row group config as the table may
have both Avro and Parquet and we may be compacting Avro, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
