rdblue commented on a change in pull request #3001:
URL: https://github.com/apache/iceberg/pull/3001#discussion_r696058734
##########
File path: flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -249,6 +251,21 @@ public Builder uidPrefix(String newPrefix) {
return this;
}
+ /**
+ * Set the {@link SlidingWindowReservoir} size (number of measurements
stored)
+ * for the two histogram metrics of data files and delete file size
distribution.
+ *
+ * @param newReservoirSize the new histogram reservoir size for the file
size distribution.
+ * default reservoir size is 128, which only add a small memory overhead
of 1 KB (128 x 8B) per histogram.
+ * For use cases with a lot of files, a larger reservoir size can produce
more accurate histogram distribution.
+ */
+ public Builder fileSizeHistogramReservoirSize(int newReservoirSize) {
Review comment:
Can we automatically detect an appropriate reservoir size based on the
number of files in a given cycle? Can we configure this at the Flink level
rather than on individual sinks? Or can we just set it high enough that we
don't care? 1KB of overhead is tiny so I doubt we care much.
My concern is that this is a public API and adding a method to it that is
this specific and obscure seems like we're going to regret adding this later
rather than coming up with a way to avoid it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
