rdblue commented on a change in pull request #3959:
URL: https://github.com/apache/iceberg/pull/3959#discussion_r805435487
##########
File path: core/src/main/java/org/apache/iceberg/MetricsConfig.java
##########
@@ -94,15 +96,21 @@ public static MetricsConfig getDefault() {
**/
@Deprecated
public static MetricsConfig fromProperties(Map<String, String> props) {
- return from(props, null);
+ return from(props, null, DEFAULT_WRITE_METRICS_MODE_DEFAULT);
}
/**
* Creates a metrics config from a table.
* @param table iceberg table
*/
public static MetricsConfig forTable(Table table) {
- return from(table.properties(), table.sortOrder());
+ String defaultMode;
+ if (table.schema().columns().size() <= MAX_COLUMNS) {
Review comment:
I think we may already have what we need to do this. Right now, the
default metrics mode is set by `write.metadata.metrics.default`. I'd propose
the following:
* If `write.metadata.metrics.default` explicitly set in table properties,
always use it
* If `write.metadata.metrics.default` is not set, check the number of columns
* If `numCols < 32` then use `truncate[16]`, the current default
* If `numCols >= 32` then use `none`
That seems like a reasonable way to make this customizable.
##########
File path: core/src/main/java/org/apache/iceberg/MetricsConfig.java
##########
@@ -94,15 +96,21 @@ public static MetricsConfig getDefault() {
**/
@Deprecated
public static MetricsConfig fromProperties(Map<String, String> props) {
- return from(props, null);
+ return from(props, null, DEFAULT_WRITE_METRICS_MODE_DEFAULT);
}
/**
* Creates a metrics config from a table.
* @param table iceberg table
*/
public static MetricsConfig forTable(Table table) {
- return from(table.properties(), table.sortOrder());
+ String defaultMode;
+ if (table.schema().columns().size() <= MAX_COLUMNS) {
Review comment:
I think this is okay for now. Tables will still respect whatever is set
as `write.metadata.metrics.default` so this really just changes Iceberg's
default in a reasonable way. It is also good to note that metrics for sort
columns are automatically promoted to at least `truncate[16]` so it isn't as
though we're losing _all_ stats.
##########
File path: core/src/main/java/org/apache/iceberg/MetricsConfig.java
##########
@@ -127,24 +135,35 @@ public static MetricsConfig forPositionDelete(Table
table) {
return new MetricsConfig(columnModes.build(), defaultMode);
}
- private static MetricsConfig from(Map<String, String> props, SortOrder
order) {
+ /**
+ * Generate a MetricsConfig for all columns based on overrides, sortOrder,
and defaultMode.
+ * @param props will be read for metrics overrides
(write.metadata.metrics.column.*) and default
+ * (write.metadata.metrics.default)
+ * @param order sort order columns, will be promoted to truncate(16)
+ * @param defaultMode default, if not set by user property
+ * @return metrics configuration
+ */
+ private static MetricsConfig from(Map<String, String> props, SortOrder
order, String defaultMode) {
Review comment:
Instead of renaming the variable that is used everywhere, I'd just
rename the incoming argument. There would be fewer changes if this used
`defaultDefaultMode` or something.
##########
File path: core/src/main/java/org/apache/iceberg/MetricsConfig.java
##########
@@ -94,15 +96,21 @@ public static MetricsConfig getDefault() {
**/
@Deprecated
public static MetricsConfig fromProperties(Map<String, String> props) {
- return from(props, null);
+ return from(props, null, DEFAULT_WRITE_METRICS_MODE_DEFAULT);
}
/**
* Creates a metrics config from a table.
* @param table iceberg table
*/
public static MetricsConfig forTable(Table table) {
- return from(table.properties(), table.sortOrder());
+ String defaultMode;
+ if (table.schema().columns().size() <= MAX_COLUMNS) {
+ defaultMode = DEFAULT_WRITE_METRICS_MODE_DEFAULT;
+ } else {
+ defaultMode = MetricsModes.None.get().toString();
+ }
+ return from(table.properties(), table.sortOrder(), defaultMode);
Review comment:
Nit: missing newline after control flow block and before `return`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
