maomaodev commented on code in PR #7476:
URL: https://github.com/apache/kyuubi/pull/7476#discussion_r3811112375
##########
extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveCatalogSuite.scala:
##########
@@ -554,4 +553,40 @@ class HiveCatalogSuite extends KyuubiHiveTest {
}
}
}
+
+ test("KyuubiParquetScan and KyuubiOrcScan hadoopConf snapshots at catalog
first use") {
+ // Unlike Spark's built-in ScanBuilders rebuilt per query planning from
+ // the current sqlConf, KSHC clones a `lazy val` snapshot taken at the
+ // catalog's first use, so confs SET after that snapshot cannot reach
+ // readers that consult the Hadoop `Configuration` directly.
+ val fieldIdRead = SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key
+ val fieldIdIgnoreMissing = SQLConf.IGNORE_MISSING_PARQUET_FIELD_ID.key
+
+ Seq("orc", "parquet").foreach { provider =>
+ withSparkSession() { spark =>
+ // `catalog` is built in `beforeEach`, its `hadoopConf` snapshot is
+ // already frozen by the time this test runs, so any conf we SET below
+ // is "mid-session" from its perspective.
+ val props = new util.HashMap[String, String]()
+ props.put(TableCatalog.PROP_PROVIDER, provider)
+
+ try {
+ val table = catalog.createTable(testIdent, schema,
Array.empty[Transform], props)
+ spark.sessionState.conf.setConfString(fieldIdRead, "true")
+ spark.sessionState.conf.setConfString(fieldIdIgnoreMissing, "true")
+
+ val hadoopConf = table.asInstanceOf[HiveTable]
+ .newScanBuilder(CaseInsensitiveStringMap.empty()).build() match {
+ case s: KyuubiParquetScan => s.hadoopConf
+ case s: KyuubiOrcScan => s.hadoopConf
+ case other => fail(s"unexpected scan type:
${other.getClass.getName}")
+ }
+ assert(hadoopConf.get(fieldIdRead) != "true")
Review Comment:
Not deliberate, added `HiveTableCatalog#newScanHadoopConf()` (non-cached,
same steps incl. token signature), both ScanBuilders clone from it, lazy val
kept for the catalog side, and flipped the test to a positive assertion on the
lowercased overlay key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
