PHILO-HE commented on code in PR #10603:
URL:
https://github.com/apache/incubator-gluten/pull/10603#discussion_r2317895302
##########
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -1172,68 +1128,54 @@ object GlutenConfig {
val COLUMNAR_MEMORY_BACKTRACE_ALLOCATION =
buildConf("spark.gluten.memory.backtrace.allocation")
- .internal()
- .doc("Print backtrace information for large memory allocations. This
helps debugging when " +
- "Spark OOM happens due to large acquire requests.")
+ .doc(
+ "Print backtrace information for large memory allocations. This helps
debugging when " +
+ "Spark OOM happens due to large acquire requests.")
.booleanConf
.createWithDefault(false)
val TRANSFORM_PLAN_LOG_LEVEL =
- buildConf("spark.gluten.sql.transform.logLevel")
- .internal()
- .stringConf
+ buildConf("spark.gluten.sql.transform.logLevel").stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValue(
logLevel => Set("TRACE", "DEBUG", "INFO", "WARN",
"ERROR").contains(logLevel),
"Valid values are 'trace', 'debug', 'info', 'warn' and 'error'.")
.createWithDefault("DEBUG")
val SUBSTRAIT_PLAN_LOG_LEVEL =
- buildConf("spark.gluten.sql.substrait.plan.logLevel")
- .internal()
- .stringConf
+ buildConf("spark.gluten.sql.substrait.plan.logLevel").stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValue(
logLevel => Set("TRACE", "DEBUG", "INFO", "WARN",
"ERROR").contains(logLevel),
"Valid values are 'trace', 'debug', 'info', 'warn' and 'error'.")
.createWithDefault("DEBUG")
val VALIDATION_LOG_LEVEL =
- buildConf("spark.gluten.sql.validation.logLevel")
- .internal()
- .stringConf
+ buildConf("spark.gluten.sql.validation.logLevel").stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValue(
logLevel => Set("TRACE", "DEBUG", "INFO", "WARN",
"ERROR").contains(logLevel),
"Valid values are 'trace', 'debug', 'info', 'warn' and 'error'.")
.createWithDefault("WARN")
val VALIDATION_PRINT_FAILURE_STACK =
- buildConf("spark.gluten.sql.validation.printStackOnFailure")
- .internal()
- .booleanConf
+ buildConf("spark.gluten.sql.validation.printStackOnFailure").booleanConf
.createWithDefault(false)
val VALIDATION_FAIL_FAST =
- buildConf("spark.gluten.sql.validation.failFast")
- .internal()
- .booleanConf
+ buildConf("spark.gluten.sql.validation.failFast").booleanConf
.createWithDefault(true)
val SOFT_AFFINITY_LOG_LEVEL =
- buildConf("spark.gluten.soft-affinity.logLevel")
- .internal()
- .stringConf
+ buildConf("spark.gluten.soft-affinity.logLevel").stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValue(
logLevel => Set("TRACE", "DEBUG", "INFO", "WARN",
"ERROR").contains(logLevel),
"Valid values are 'trace', 'debug', 'info', 'warn' and 'error'.")
.createWithDefault("DEBUG")
val DEBUG_ENABLED =
- buildConf("spark.gluten.sql.debug")
- .internal()
- .booleanConf
+ buildConf("spark.gluten.sql.debug").booleanConf
Review Comment:
I agree with exposing only public configs in Gluten document. However, we
need to establish a convention to guide developers on how to categorize configs.
It seems Spark has no official guide to categorize a config to either
internal or public. I note debug-related configs are marked as internal,
including debug enabling, debug level setting. And some experimental configs
are also marked as internal. I think we can follow such convention.
According to my understanding, public configs are those exposed to end users
and they are already stable to use. And internal configs are those for
developers or advanced users to use.
Please add a comment for Gluten's `internal()` method to clarify, and review
all Gluten configs to see if `internal` is correctly used or removed.
cc @zhouyuan @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
