rdblue commented on a change in pull request #3977:
URL: https://github.com/apache/iceberg/pull/3977#discussion_r794846901
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -210,6 +211,20 @@ private TableProperties() {
public static final String SPARK_WRITE_PARTITIONED_FANOUT_ENABLED =
"write.spark.fanout.enabled";
public static final boolean SPARK_WRITE_PARTITIONED_FANOUT_ENABLED_DEFAULT =
false;
+ public static final String PARTITIONED_FANOUT_WRITERS_CACHE_SIZE =
"write.partition.fanout.writers-cache-size";
+ public static final int PARTITIONED_FANOUT_WRITERS_CACHE_SIZE_DEFAULT =
Integer.MAX_VALUE;
+
+ public static final String PARTITIONED_FANOUT_WRITERS_CACHE_EVICT_MS =
+ "write.partition.fanout.writers-cache-eviction-ms";
+ public static final long PARTITIONED_FANOUT_WRITERS_CACHE_EVICT_MS_DEFAULT =
TimeUnit.MINUTES.toMillis(5);
Review comment:
What is the purpose of a timeout? I don't see how that aligns with the
need to reduce memory consumption. You could create so many files in 5 minutes
that you run out of memory. If you could evict based on memory consumption of
the writers, that seems valuable. But I think I would remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
