andygrove opened a new issue, #6635:
URL: https://github.com/apache/datafusion-comet/issues/6635

   Part of #6615.
   
   ## What / Why
   
   This issue covers the remaining smaller expression areas:
   
   - conditional and predicates: `CometIfSuite`, `CometCaseWhenSuite`, 
`CometCoalesceSuite`, and `IN` / `InSet` / `<=>` / boolean tests in 
`CometExpressionSuite`;
   - JSON: `CometJsonExpressionSuite`, `CometJsonJvmSuite`;
   - CSV: `CometCsvExpressionSuite`;
   - `uuid`: `CometUuidExpressionSuite`;
   - `width_bucket` enable/disable: both `CometWidthBucketSuite` copies;
   - Variant: `CometVariantProjectionSuite`;
   - nondeterministic functions (`rand`, `randn`, `spark_partition_id`, 
`monotonically_increasing_id`) and fallback-reason plumbing in 
`CometExpressionSuite`.
   
   It also accounts for several serde-level suites that stay in Scala: 
`CometLiteralSuite`, `CometScalarFunctionSuite`, `CometToPrettyStringSuite`, 
`CometSparkInternalFunctionsSuite` and `CometVariantTypeSuite`.
   
   Static analysis on `main` at 3bc2faa934 found the following:
   
   | Suite | Covered | Convert | Blocked | Split | Keep | Total |
   |---|---|---|---|---|---|---|
   | `CometNativeCastSuite` | 0 | 1 | 0 | 0 | 0 | 1 |
   | `CometExpressionSuite` | 3 | 13 | 1 | 1 | 14 | 32 |
   | `CometJsonExpressionSuite` | 1 | 7 | 0 | 0 | 0 | 8 |
   | `CometJsonJvmSuite` | 1 | 2 | 0 | 0 | 0 | 3 |
   | `CometCsvExpressionSuite` | 0 | 2 | 0 | 0 | 0 | 2 |
   | `CometUuidExpressionSuite` | 0 | 0 | 0 | 2 | 0 | 2 |
   | `CometIfSuite` | 0 | 1 | 0 | 0 | 0 | 1 |
   | `CometCaseWhenSuite` | 0 | 1 | 0 | 0 | 0 | 1 |
   | `CometCoalesceSuite` | 0 | 3 | 0 | 0 | 0 | 3 |
   | `CometWidthBucketSuite (spark-3.5)` | 0 | 1 | 0 | 0 | 0 | 1 |
   | `CometWidthBucketSuite (spark-4.x)` | 0 | 1 | 0 | 0 | 0 | 1 |
   | `CometToPrettyStringSuite (spark-3.5)` | 0 | 0 | 0 | 0 | 2 | 2 |
   | `CometToPrettyStringSuite (spark-4.x)` | 0 | 0 | 0 | 0 | 2 | 2 |
   | `CometSparkInternalFunctionsSuite (spark-3.5)` | 0 | 0 | 0 | 0 | 1 | 1 |
   | `CometSparkInternalFunctionsSuite (spark-4.x)` | 0 | 0 | 0 | 0 | 1 | 1 |
   | `CometLiteralSuite` | 0 | 0 | 2 | 0 | 2 | 4 |
   | `CometScalarFunctionSuite` | 0 | 0 | 0 | 0 | 14 | 14 |
   | `CometVariantTypeSuite` | 0 | 0 | 0 | 0 | 1 | 1 |
   | `CometVariantProjectionSuite` | 0 | 6 | 0 | 1 | 6 | 13 |
   | **Total** | **5** | **38** | **3** | **4** | **43** | **93** |
   
   ## Work
   
   Line numbers refer to `CometExpressionSuite` unless a suite is named.
   
   **Conditional and predicates**
   
   - `conditional/if_expr.sql` (`CometIfSuite` L27): `IF(c1 < 2, 1111, 2222)` 
(int-literal branches over a column predicate), `IF(c1 < c3, ...)`, and the 
string variant.
   - `conditional/case_when.sql` (`CometCaseWhenSuite` L27)
     - The six int-literal CASE shapes.
     - The `WHEN NULL` branch is simplified away in both suites, so there's 
nothing to keep for it.
   - `conditional/coalesce.sql` (`CometCoalesceSuite` L29, L54)
     - `coalesce(cast(t1 AS date), cast(t2 AS date), d)`, the #168 shape.
     - String coalesce. Only `coalesce(s1, 1)` and `coalesce(s2, s1)` survive 
`NullPropagation`; the other four Scala expressions never reached Comet.
   - `conditional/case_when_lazy.sql` (`CometCoalesceSuite` L42): `coalesce(a, 
CAST(a AS TINYINT))`. The file's ANSI ConfigMatrix supplies the ANSI leg.
   - `conditional/case_when_dictionary.sql` (new, `CometNativeCastSuite` L2540, 
#2038)
     - `parquet.enable.dictionary=true`, and `INSERT INTO t SELECT id < 5000, 
'one' FROM range(10000)`.
     - Query: `CASE WHEN (b = true) THEN s ELSE '' END`.
   - `conditional/in_set.sql` + `conditional/in_set_inset.sql` (new) (L2110)
     - Split by the inSet threshold so queries can use `expect_native(in)` and 
`expect_native(inset)`, plus a ConfigMatrix over the codegen flag.
     - Today `in_set.sql` uses a plain `query` where the Scala test pins the 
mechanism in both codegen modes.
   - `conditional/in_nested_zero.sql` (L505)
     - A scalar `a IN (b, 13.0D)` with signed zeros, using `expect_native(in)` 
under `filterPushdown=false`.
     - Fix the file's InSet leg. Its `array(CAST(...))` candidates aren't 
Literals once ConstantFolding is excluded, so the `inSetConversionThreshold=0` 
run never produces InSet, despite its comment.
   - `conditional/in_float_signed_zero.sql` + `inset_float_signed_zero.sql` 
(new; L447, convertible half): ±0.0 and NULL literal candidates (`-0.0F`, 
`-0.0D`).
   - `conditional/predicates.sql` (L2086): the eight `IS [NOT] TRUE/FALSE` and 
`[NOT] _2 <=> true/false` WHERE predicates over a nullable boolean column.
   - `conditional/deep_boolean_chains.sql` (new, L3828)
     - 200-term AND and OR chains, written with explicit nested parentheses. 
Spark's parser rebalances AND/OR chains that have none.
     - Check that 200 levels of parentheses parse.
   
   **JSON and CSV**
   
   - `struct/structs_to_json.sql` (`CometJsonExpressionSuite` L51)
     - Add ordinary fractional, Min/Max and -0.0 floats and doubles. The 
fixture's float columns hold only NaN, ±Infinity, 0.0 and NULL.
     - Add strings with control and supplementary characters.
   - `struct/json_to_structs_native.sql` (new, 
`JsonToStructs.allowIncompatible=true`): `CometJsonExpressionSuite` L88-L188.
     - Basic and all-primitive schemas.
     - Null and error rows: missing field, explicit null, invalid JSON, `{}`, 
`null` and NULL.
     - `IS NULL` and field access on a NULL input, a nested struct schema, and 
type-mismatch rows.
     - **No fixture reaches native `from_json` today.**
   - `struct/json_to_structs.sql` (`CometJsonJvmSuite` L59, L65)
     - Switch lines 24 and 28 from `spark_answer_only` to `query`, or to 
`expect_dispatch(from_json)`. They date from #3328, and `from_json` became 
dispatched by default in #4305.
     - Add `to_json(from_json(j, 'a INT, b STRING'))`.
   - `csv/to_csv.sql` (`CometCsvExpressionSuite` L36)
     - A 13-column table: boolean through decimal(36,18), date, timestamp, 
timestamp_ntz and string.
     - Generator-style edges: integral min/max, NaN, ±Infinity, -0.0, a 
36-digit decimal and a far-future timestamp.
   - `csv/to_csv_native_options.sql` (new, 
`StructsToCsv.allowIncompatible=true`; `CometCsvExpressionSuite` L68)
     - The 17 tricky strings and the 18 option maps, with 
`expect_native(to_csv)` on at least one query.
     - Reading the code, native `to_csv` appears to ignore the `sep` alias: 
`CometStructsToCsv.options2Proto` reads only `delimiter`. Check this while 
converting, and file it if it reproduces.
   
   **Nondeterministic functions and misc**
   
   - `misc/rand.sql` (L3660, L3672, L3704)
     - `rand` and `randn` with seeds 42, `Long.MaxValue` and NULL, over 
`REPARTITION(n)` subqueries, with `spark.comet.batchSize=7`.
     - Use `tolerance=1e-6` for `randn`.
   - `misc/spark_partition_id.sql` (L3684) and 
`misc/monotonically_increasing_id.sql` (L3694): two-level repartition shapes, 
with the expression in the outer query.
   - For all five, load the input with `INSERT INTO t SELECT id FROM range(0, 
100, 1, 1)` so it is one file. `INSERT ... VALUES` writes one file per row and 
changes the input partitions.
   - `misc/null_literals.sql` (new, L665): typed NULL struct, map and array 
literals, plus arithmetic with NULL over int and decimal columns.
   - `misc/uuid_with_seed.sql` (`CometUuidExpressionSuite` L43, L52)
     - The 4.0+ halves are already covered at lines 80-92.
     - For the multi-partition case, use `/*+ REPARTITION(4, id) */` instead of 
`DISTRIBUTE BY id`. The fixture's existing lines 65-66 are probably vacuous: 
AQE coalesces `REPARTITION_BY_COL`, and 32 tiny rows likely end up in one 
partition.
     - The 3.4/3.5 halves (SQL `uuid(seed)` only exists from 4.0) and the 
`getNumPartitions > 1` assertion stay in Scala.
   - `misc/width_bucket_disabled.sql` (new; both `CometWidthBucketSuite` 
copies, which are identical)
     - Set `spark.comet.expression.WidthBucket.enabled=false`.
     - Assert `expect_fallback(spark.comet.expression.WidthBucket.enabled=true 
to enable it)`.
   - **Fallback-reason plumbing**
     - `misc/expression_disabled_config.sql` (L1933): 
`spark.comet.expression.Add.enabled=false` with `expect_fallback(Expression 
support is disabled...)`.
     - `misc/fallback_reasons_shuffle_disabled.sql` (L2522): 
`spark.comet.shuffle.enabled=false` with four `expect_fallback` queries.
     - `misc/fallback_reason_collect_limit_disabled.sql` (L2566).
   
   **Variant** (`CometVariantProjectionSuite`, all `MinSparkVersion: 4.0`)
   
   - `misc/variant_native_scan.sql` (new): L88, L104, L233, L257, and the 
answer half of L295.
     - Set `allowReadingShredded=true` and `pushVariantIntoScan=false`.
     - L104 needs `ConfigMatrix writeShredding=false,true`.
     - L233 uses `ALTER TABLE ADD COLUMN ... DEFAULT`.
   - Three single-config files for L273: `nanosAsLong=true`, 
`inferTimestampNTZ=false` and `pushVariantIntoScan=true`.
   - `misc/variant_encrypted_scan_fallback.sql` (new, L340)
     - Use the Parquet encryption configs.
     - The `Config` directive parser splits on the last `=`, so a base64 key 
ending in `==` cannot be written. Use an unpadded key, or fix the parser in 
#6618.
   
   **Blocked**
   
   - **#6618, ConstantFolding:**
     - L64: nested float `IN` candidates, which must fold to array Literals.
     - The NaN-literal half of L447.
     - `CometLiteralSuite` L132 and L146, the folded array-literal path in 
`CometLiteral.makeListLiteral`.
   
   ## Stays in Scala
   
   - **Serde internals and helper unit tests:**
     - All 14 `CometScalarFunctionSuite` tests, which use synthetic 
`Unevaluable` expressions plus a registry audit.
     - `CometLiteralSuite` L96 and L120.
     - `CometVariantTypeSuite` L58.
     - L58 (`checkCometAnswer` labels), L1383 (`withInfo`), L1420 
(`withFallbackReason`) and L3975 (`NativeOptIn`).
   - **Explain / COMET-INFO text:** L1442, L3986, L4005, L4016.
   - **Internal expressions SQL can't reach:**
     - `ToPrettyString`, 4 tests. Spark builds it only in `Dataset.show`.
     - `Empty2Null`, 2 tests. Only planned writes insert it.
   - **DataFrame-only inputs:**
     - L2143: an empty `IN` list and a programmatic `InSet(Set.empty)`.
     - L3719: `DataFrameReader.schema` with fields missing from the file.
   - **UDF registration:** L3900, which also inspects the exception chain.
   - **Raw NaN payload bits:** L402.
   - **Raw-writer Parquet layouts:**
     - L631 and L647: UINT_32/64, ENUM, FLBA, and decimals stored as 
INT32/INT64/FLBA.
     - `CometVariantProjectionSuite` L118, L144, L179, L245, L354: hand-built 
shredded Variant structs, case-folded physical names, and a raw writer with 
required value fields.
   - **`CometVariantProjectionSuite` L206:** reader-schema existence defaults 
combined with a serde assertion.
   
   ## Notes on the Scala tests
   
   - **L631's dictionary run returns no rows.**
   - **L693's date-literal filter never matches.** That test is in the datetime 
issue.
   
   ## Done when
   
   - Every test in the table below is accounted for. Either a fixture covers it 
(the same inputs, configs and assertion strength, or stronger), the fixture 
that already covers it is cited, or it stays in Scala for the reason given 
above.
   - Each converted Scala test is deleted in the PR that adds its coverage. A 
suite left empty is deleted and removed from the suite lists in 
`.github/workflows/pr_build_linux.yml` and `pr_build_macos.yml`.
   - The new and changed fixtures pass on every Spark profile. Apply 
`run-all-spark-profiles`, because pull requests run only the default profile.
   
   <details><summary>Every Scala test in scope, with its verdict and target 
fixture</summary>
   
   | Suite | Line | Test | Verdict | Target |
   |---|---|---|---|---|
   | `CometCaseWhenSuite` | 27 | case_when | convert | 
conditional/case_when.sql |
   | `CometCoalesceSuite` | 29 | coalesce should return correct datatype | 
convert | conditional/coalesce.sql |
   | `CometCoalesceSuite` | 42 | test coalesce lazy eval | convert | 
conditional/case_when_lazy.sql |
   | `CometCoalesceSuite` | 54 | string with coalesce | convert | 
conditional/coalesce.sql |
   | `CometCsvExpressionSuite` | 36 | to_csv - default options | convert | 
csv/to_csv.sql |
   | `CometCsvExpressionSuite` | 68 | to_csv - with configurable formatting 
options | convert | csv/to_csv_native_options.sql (new) |
   | `CometExpressionSuite` | 58 | check output labels on mismatch (ignored) | 
keep |  |
   | `CometExpressionSuite` | 64 | nested floating point membership uses native 
In and InSet | blocked: #6618 | conditional/in_nested_zero.sql |
   | `CometExpressionSuite` | 370 | compare true/false to negative zero | 
covered | conditional/boolean.sql |
   | `CometExpressionSuite` | 402 | compare $dataType columns with noncanonical 
NaN literals (2 instances: float/_1, double/_2) | keep |  |
   | `CometExpressionSuite` | 447 | floating $name and NOT $name normalize NaNs 
and signed zeros (2 instances: In with threshold 10, InSet with threshold 1) | 
split (rest: #6618) | conditional/in_float_signed_zero.sql (new, threshold 10) 
+ conditional/inset_float_signed_zero.sql (new, threshold 1) |
   | `CometExpressionSuite` | 505 | floating IN with a column candidate 
normalizes signed zeros | convert | conditional/in_nested_zero.sql |
   | `CometExpressionSuite` | 537 | parquet default values | covered | 
misc/parquet_default_values.sql |
   | `CometExpressionSuite` | 631 | basic data type support - excluding u8/u16 
| keep |  |
   | `CometExpressionSuite` | 647 | uint data type support - excluding u8/u16 | 
keep |  |
   | `CometExpressionSuite` | 665 | null literals | convert | 
misc/null_literals.sql (new) |
   | `CometExpressionSuite` | 1383 | withInfo records non-fallback info 
rendered as COMET-INFO | keep |  |
   | `CometExpressionSuite` | 1420 | withFallbackReason | keep |  |
   | `CometExpressionSuite` | 1442 | native opt-in hint shown for 
codegen-dispatch path | keep |  |
   | `CometExpressionSuite` | 1933 | disable expression using dynamic config | 
convert | misc/expression_disabled_config.sql (new) |
   | `CometExpressionSuite` | 2086 | EqualNullSafe should preserve comet filter 
| convert | conditional/predicates.sql |
   | `CometExpressionSuite` | 2110 | test in(set)/not in(set) | convert | 
conditional/in_set.sql + conditional/in_set_inset.sql (new) |
   | `CometExpressionSuite` | 2143 | null IN empty list honours legacy 
null-in-empty behavior | keep |  |
   | `CometExpressionSuite` | 2191 | not | covered | conditional/boolean.sql |
   | `CometExpressionSuite` | 2522 | explain comet | convert | 
misc/fallback_reasons_shuffle_disabled.sql (new) |
   | `CometExpressionSuite` | 2566 | explain: CollectLimit disabled | convert | 
misc/fallback_reason_collect_limit_disabled.sql (new) |
   | `CometExpressionSuite` | 3660 | rand expression with random parameters | 
convert | misc/rand.sql (new) |
   | `CometExpressionSuite` | 3672 | randn expression with random parameters | 
convert | misc/rand.sql (new) |
   | `CometExpressionSuite` | 3684 | spark_partition_id expression on random 
dataset | convert | misc/spark_partition_id.sql (new) |
   | `CometExpressionSuite` | 3694 | monotonically_increasing_id expression on 
random dataset | convert | misc/monotonically_increasing_id.sql (new) |
   | `CometExpressionSuite` | 3704 | multiple nondetermenistic expressions with 
shuffle | convert | misc/rand.sql (new) |
   | `CometExpressionSuite` | 3719 | vectorized reader: missing all struct 
fields | keep |  |
   | `CometExpressionSuite` | 3828 | deep AND/OR predicate chains do not 
overflow the protobuf recursion limit | convert | 
conditional/deep_boolean_chains.sql (new) |
   | `CometExpressionSuite` | 3900 | pushed-down predicate divide-by-zero is 
not surfaced as FAILED_READ_FILE | keep |  |
   | `CometExpressionSuite` | 3975 | NativeOptIn message and Compatible field | 
keep |  |
   | `CometExpressionSuite` | 3986 | RLike out-of-subset literal shows native 
opt-in, in-subset and non-literal do not | keep |  |
   | `CometExpressionSuite` | 4005 | upper shows native opt-in via 
caseConversion config key | keep |  |
   | `CometExpressionSuite` | 4016 | date_format shows native opt-in for 
whitelisted literal format | keep |  |
   | `CometIfSuite` | 27 | if expression | convert | conditional/if_expr.sql |
   | `CometJsonExpressionSuite` | 51 | to_json - all supported types | convert 
| struct/structs_to_json.sql |
   | `CometJsonExpressionSuite` | 75 | to_json - options and unsupported native 
types route to the codegen engine | covered | struct/structs_to_json.sql |
   | `CometJsonExpressionSuite` | 88 | from_json - basic primitives | convert | 
struct/json_to_structs_native.sql (new) |
   | `CometJsonExpressionSuite` | 106 | from_json - null and error handling | 
convert | struct/json_to_structs_native.sql (new) |
   | `CometJsonExpressionSuite` | 128 | from_json - all primitive types | 
convert | struct/json_to_structs_native.sql (new) |
   | `CometJsonExpressionSuite` | 148 | from_json - null input produces null 
struct | convert | struct/json_to_structs_native.sql (new) |
   | `CometJsonExpressionSuite` | 169 | from_json - nested struct | convert | 
struct/json_to_structs_native.sql (new) |
   | `CometJsonExpressionSuite` | 188 | from_json - valid json with 
incompatible schema | convert | struct/json_to_structs_native.sql (new) |
   | `CometJsonJvmSuite` | 52 | get_json_object via JVM engine | covered | 
string/routing_string_opt_in_enabled.sql |
   | `CometJsonJvmSuite` | 59 | from_json with explicit schema via JVM engine | 
convert | struct/json_to_structs.sql |
   | `CometJsonJvmSuite` | 65 | to_json round-trip via JVM engine | convert | 
struct/json_to_structs.sql |
   | `CometLiteralSuite` | 96 | listLiteralElementSupported mirrors 
makeListLiteral's arms, flat and nested | keep |  |
   | `CometLiteralSuite` | 120 | a non-default collation is declined on both 
sides | keep |  |
   | `CometLiteralSuite` | 132 | a year-month interval array literal is 
serialized rather than declined | blocked: #6618 | 
array/array_literal_folded.sql (new, needs ConstantFolding) |
   | `CometLiteralSuite` | 146 | a folded nested array literal with an empty 
branch and non-nullable elements | blocked: #6618 | 
array/array_literal_folded.sql (new, needs ConstantFolding) |
   | `CometNativeCastSuite` | 2540 | test implicit cast to dictionary with case 
when and dictionary type | convert | conditional/case_when_dictionary.sql (new) 
|
   | `CometScalarFunctionSuite` | 103 | CometScalarFunction rejects 
ANSI-sensitive expressions (#5074) | keep |  |
   | `CometScalarFunctionSuite` | 110 | rejects expressions with failOnError 
even when false (#5074) | keep |  |
   | `CometScalarFunctionSuite` | 117 | isAnsiSensitive detects failOnError 
field | keep |  |
   | `CometScalarFunctionSuite` | 124 | isAnsiSensitive detects evalMode field 
| keep |  |
   | `CometScalarFunctionSuite` | 132 | isAnsiSensitive detects nullOnOverflow 
field | keep |  |
   | `CometScalarFunctionSuite` | 140 | isAnsiSensitive detects ansiEnabled 
field | keep |  |
   | `CometScalarFunctionSuite` | 148 | isAnsiSensitive detects evalContext 
field | keep |  |
   | `CometScalarFunctionSuite` | 154 | isAnsiSensitive detects Spark Round 
ansiEnabled | keep |  |
   | `CometScalarFunctionSuite` | 165 | rejects expressions with evalMode via 
plain CometScalarFunction (#5074) | keep |  |
   | `CometScalarFunctionSuite` | 180 | rejects expressions with nullOnOverflow 
via plain CometScalarFunction (#5074) | keep |  |
   | `CometScalarFunctionSuite` | 195 | rejects expressions with ansiEnabled 
via plain CometScalarFunction | keep |  |
   | `CometScalarFunctionSuite` | 210 | rejects expressions with evalContext 
via plain CometScalarFunction | keep |  |
   | `CometScalarFunctionSuite` | 219 | CometScalarFunction allows non-ANSI 
expressions | keep |  |
   | `CometScalarFunctionSuite` | 231 | no ANSI-sensitive expression is 
registered with plain CometScalarFunction | keep |  |
   | `CometSparkInternalFunctionsSuite (spark-3.5)` | 27 | empty2null is 
offloaded to Comet | keep |  |
   | `CometSparkInternalFunctionsSuite (spark-4.x)` | 28 | empty2null is 
offloaded to Comet | keep |  |
   | `CometToPrettyStringSuite (spark-3.5)` | 36 | ToPrettyString | keep |  |
   | `CometToPrettyStringSuite (spark-3.5)` | 59 | ToPrettyString honors 
spark.comet.expression.ToPrettyString.enabled | keep |  |
   | `CometToPrettyStringSuite (spark-4.x)` | 39 | ToPrettyString | keep |  |
   | `CometToPrettyStringSuite (spark-4.x)` | 71 | ToPrettyString honors 
spark.comet.expression.ToPrettyString.enabled | keep |  |
   | `CometUuidExpressionSuite` | 43 | seeded uuid matches Spark bit for bit | 
split | misc/uuid_with_seed.sql |
   | `CometUuidExpressionSuite` | 52 | seeded uuid matches Spark bit for bit 
across multiple partitions | split (rest: plan shape stays in Scala) | 
misc/uuid_with_seed.sql |
   | `CometVariantProjectionSuite` | 88 | direct Variant projection preserves 
values and siblings | convert | misc/variant_native_scan.sql (new) |
   | `CometVariantProjectionSuite` | 104 | Variant objects with empty keys 
match Spark | convert | misc/variant_native_scan.sql (new) |
   | `CometVariantProjectionSuite` | 118 | shredded Variant missing fields and 
redundant residuals match Spark | keep |  |
   | `CometVariantProjectionSuite` | 144 | shredded Variant scalar encodings 
and dictionary traversal match Spark bytes | keep |  |
   | `CometVariantProjectionSuite` | 179 | malformed shredded Variant values 
report Spark's error class | keep |  |
   | `CometVariantProjectionSuite` | 206 | non-null Variant existence defaults 
fall back to Spark | keep |  |
   | `CometVariantProjectionSuite` | 233 | null Variant existence defaults 
preserve later default indexes | convert | misc/variant_native_scan.sql (new) |
   | `CometVariantProjectionSuite` | 245 | Variant projection uses shared 
Unicode field matching | keep |  |
   | `CometVariantProjectionSuite` | 257 | unread Variant roots and nested 
fields are pruned from native scans | convert | misc/variant_native_scan.sql 
(new) |
   | `CometVariantProjectionSuite` | 273 | Variant scans preserve strict reader 
and timestamp inference fallbacks | convert | 
misc/variant_scan_legacy_nanos_fallback.sql + 
variant_scan_ntz_inference_fallback.sql + variant_push_into_scan_fallback.sql 
(new) |
   | `CometVariantProjectionSuite` | 295 | Variant consumers fall back above a 
native scan | split (rest: plan shape stays in Scala) | 
misc/variant_native_scan.sql (new) |
   | `CometVariantProjectionSuite` | 340 | encrypted Variant scans fall back to 
Spark | convert | misc/variant_encrypted_scan_fallback.sql (new) |
   | `CometVariantProjectionSuite` | 354 | strict Variant reader preserves 
malformed layout errors | keep |  |
   | `CometVariantTypeSuite` | 58 | Variant identity requires the canonical 
Arrow extension marker | keep |  |
   | `CometWidthBucketSuite (spark-3.5)` | 30 | WidthBucket honors 
spark.comet.expression.WidthBucket.enabled | convert | 
misc/width_bucket_disabled.sql (new) |
   | `CometWidthBucketSuite (spark-4.x)` | 30 | WidthBucket honors 
spark.comet.expression.WidthBucket.enabled | convert | 
misc/width_bucket_disabled.sql (new) |
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to