Copilot commented on code in PR #12549:
URL: https://github.com/apache/gluten/pull/12549#discussion_r3711195306
##########
gluten-substrait/src/main/scala/org/apache/spark/shuffle/GlutenShuffleUtils.scala:
##########
@@ -49,42 +48,36 @@ object GlutenShuffleUtils {
}
def getCompressionCodec(conf: SparkConf): String = {
- def checkCodecValues(codecConf: String, codec: String, validValues:
Set[String]): Unit = {
+ // Gluten's codec conf falls back to Spark's spark.io.compression.codec,
so reading it always
+ // yields a value. Look both keys up in SQLConf first and then in the
given SparkConf, matching
+ // how a session inherits from the application conf.
+ val provider =
+ new ChainedProvider(new SQLConfProvider(SQLConf.get), new
SparkConfProvider(conf))
+ val codecEntry = GlutenConfig.COLUMNAR_SHUFFLE_CODEC
+ val (codec, isSetOnGlutenConf) = codecEntry.readWithSource(provider)
+ val supportedCodecs =
BackendsApiManager.getSettings.shuffleSupportedCodec()
+ if (isSetOnGlutenConf) {
+ // An explicitly set codec is validated against the codec backend in use.
+ val validValues = if (GlutenConfig.get.columnarShuffleEnableQat) {
+ GlutenConfig.GLUTEN_QAT_SUPPORTED_CODEC
+ } else {
+ supportedCodecs
+ }
if (!validValues.contains(codec)) {
throw new IllegalArgumentException(
- s"The value of $codecConf should be one of " +
+ s"The value of ${codecEntry.key} should be one of " +
s"${validValues.mkString(", ")}, but was $codec")
}
Review Comment:
`validValues` is a `Set[String]`, so `mkString` can produce nondeterministic
ordering across runs/JVMs. That makes this error message flaky and harder to
compare in tests/logs; sort before joining.
This issue also appears on line 73 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
