andygrove opened a new issue, #279:
URL: https://github.com/apache/datafusion-spark/issues/279

   Part of #273 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `schema_of_variant_agg(v)` is an aggregate. It merges the schemas of all 
non-NULL VARIANT values in the group and
   returns the result as a `STRING`. The output format is the same as 
`schema_of_variant`: Spark SQL type syntax with
   `OBJECT<...>` for objects. The argument must be `VARIANT`. The result is 
non-nullable and uses the default string
   type. Added in Spark 4.0.0.
   
   The buffer starts as `VOID` (`NullType`). Each row merges 
`SchemaOfVariant.schemaOf(v)` into it with
   `SchemaOfVariant.mergeSchema`, and partial buffers are combined the same 
way. SQL NULL rows are skipped. The merge
   and print rules are the ones in the `schema_of_variant` issue, so the two 
should share code.
   
   ### Examples
   
   ```sql
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('1'), ('2'), ('3') 
AS tab(j);
   -- BIGINT
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": 1}'), 
('{"b": true}'), ('{"c": 1.23}') AS tab(j);
   -- OBJECT<a: BIGINT, b: BOOLEAN, c: DECIMAL(3,2)>
   SELECT schema_of_variant_agg(parse_json('{"a": [1, 2, 3]}'));
   -- OBJECT<a: ARRAY<BIGINT>>
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": [1, 2, 
3]}'), ('{"a": "banana"}') AS tab(j);
   -- OBJECT<a: VARIANT>
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": "banana"}'), 
('{"b": "apple"}') AS tab(j);
   -- OBJECT<a: STRING, b: STRING>
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": "data"}'), 
(null) AS tab(j);
   -- OBJECT<a: STRING>
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES (null), (null) AS 
tab(j);
   -- VOID
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": null}'), 
('{"a": null}') AS tab(j);
   -- OBJECT<a: VOID>
   SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES
     ('{"hi":[]}'), ('{"hi":[{},{}]}'), ('{"hi":[{"it''s":[{"me":[{"a": 
1}]}]}]}') AS tab(j);
   -- OBJECT<hi: ARRAY<OBJECT<`it's`: ARRAY<OBJECT<me: ARRAY<OBJECT<a: 
BIGINT>>>>>>>
   ```
   
   These come from the `ExpressionDescription` and 
`VariantEndToEndSuite."schema_of_variant_agg"`. The suite builds
   the inputs as DataFrames, and the queries above are the equivalent SQL.
   
   ### Edge cases to test
   
   - Grouping. `VariantEndToEndSuite` uses rows `id` in 0..100 with `{"a": 
[id]}` when `id % 4 == 0` and
     `{"a": ["id"]}` otherwise:
     - `GROUP BY id % 2` gives `OBJECT<a: ARRAY<STRING>>` and `OBJECT<a: 
ARRAY<VARIANT>>`;
     - `% 3` gives `OBJECT<a: ARRAY<VARIANT>>` three times;
     - `% 4` gives `OBJECT<a: ARRAY<STRING>>` three times and `OBJECT<a: 
ARRAY<BIGINT>>` once.
     These exercise the multi-partition merge path.
   - If every input is NULL the result is `VOID`. With no input rows the buffer 
is never updated, so by code reading
     the result is also `VOID`, not NULL, because the function is non-nullable.
   - A merge across rows behaves like a merge across array elements. For 
example, `BIGINT` with `DOUBLE` gives
     `DOUBLE`, objects are unioned with sorted field names, and incompatible 
types give `VARIANT`. See the full table
     in the `schema_of_variant` issue.
   - Order independence: decimal widening, struct union and `VOID` absorption 
are commutative. One exception, by code
     reading, is struct merging through `findTightestCommonType`, which keeps 
the first field name when names are
     equal case-insensitively (`spark.sql.caseSensitive=false`). The result 
could depend on the order of rows or
     partitions, so do not test that case for determinism.
   - Field-name quoting with backticks follows `QuotingUtils.quoteIfNeeded`. 
For example, `!special!` is quoted and
     `STRUCT` is not.
   - Malformed input raises `MALFORMED_VARIANT` (unsorted object fields) or 
`UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT`, the
     same as `schema_of_variant`.
   - Test a literal argument, a column argument, `GROUP BY`, and use as a 
window aggregate if datafusion-spark
     supports that for UDAFs.
   
   ### Differences across Spark versions
   
   - 3.x has no VARIANT.
   - No differences between 4.0.4, 4.1.3 and 4.2.0.
   
   ### Implementation notes
   
   - Spark: `SchemaOfVariantAgg` (a `TypedImperativeAggregate[DataType]`) in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
 Spark
     serializes the buffer as `DataType.json`, which is internal.
   - In DataFusion, write an `AggregateUDFImpl` / `Accumulator` whose state is 
the in-progress Spark type tree. The
     state can be a single `Utf8` field, with the type serialized in any 
internal form. `merge_batch` applies the same
     `mergeSchema` as `update_batch`. Reuse the type lattice and printer from 
`schema_of_variant`.
   - DataFusion core and arrow-rs have no equivalent. Use 
`parquet_variant::Variant` to decode values. This depends on
     the VARIANT Arrow representation proposed in the `parse_json` issue.
   - apache/datafusion-comet#5438 tracks native Variant support in Comet.
   - Put it under `crates/spark/src/function/aggregate/` or the new `variant/` 
module. Add
     `crates/sqllogictest/test_files/variant/schema_of_variant_agg.slt` and 
document the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to