John Zhuge created SPARK-61843:
----------------------------------

             Summary: Follow-up tests for spark.sql.extensions.defaultList
                 Key: SPARK-61843
                 URL: https://issues.apache.org/jira/browse/SPARK-61843
             Project: Spark
          Issue Type: Task
          Components: Spark Core
    Affects Versions: 4.3.0
            Reporter: John Zhuge


Follow-up to SPARK-60483 (PR #59743), which adds 
spark.sql.extensions.defaultList. That PR pins the default list ahead of user 
extensions, the default-only case, a shared extension applied once, and 
last-win ordering with user-list repeats. These cases are not covered yet.

1. Repeats inside the default list, and the warning count when the two lists 
overlap. Every default-list test in #59743 configures a single entry, so these 
mutations pass the whole suite:
   - defaultClassNames.distinct.filterNot(userClassNames.contains(_)) drops 
repeats inside the default list. Caught by default = MyExtensions,MyExtensions 
with user = MyExtensions2, asserting MyRule runs twice.
   - defaultClassNames.diff(userClassNames) keeps default-list occurrences 
beyond the user count. Caught by default = MyExtensions,MyExtensions with user 
= MyExtensions, asserting MyRule runs once.
   - Dropping .distinct from the warning loop logs one warning per default-list 
occurrence instead of per class. Caught by the same case, counting the matching 
log messages (1, not 2); an exists() check does not see it.
   Also assert zero matching warnings when the two lists are disjoint, so 
warning once per default entry cannot pass.

2. spark.conf.getOption("spark.sql.extensions") is empty when only the default 
list is set, and returns the raw user value when spark.sql.extensions is set; 
SET spark.sql.extensions shows <undefined> before it is set and the raw user 
value after. Assert through getOption so the test does not depend on 
null-versus-throw for an unset optional entry.

3. A runtime SET spark.sql.extensions.defaultList, or spark.conf.set on it, 
throws CANNOT_MODIFY_STATIC_CONFIG. This pins the buildStaticConf registration; 
the guard itself is requireNonStaticConf, which is key-independent apart from 
its documented spark.default.parallelism carve-out.

The follow-up PR should also list these cases under "How was this patch 
tested?" (sarutak, Nov-13 inline review).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to