andygrove opened a new issue, #3160:
URL: https://github.com/apache/datafusion-comet/issues/3160
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `json_tuple` function, causing
queries using this function to fall back to Spark's JVM execution instead of
running natively on DataFusion.
JsonTuple is a generator expression that extracts multiple fields from a
JSON string and returns them as separate columns in a tuple format. It takes a
JSON string as the first argument followed by one or more field path
expressions, returning a row with the extracted values in the order specified.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
json_tuple(json_string, field1, field2, ...)
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| json_string | String | The JSON string to parse and extract fields from |
| field1, field2, ... | String | One or more field names/paths to extract
from the JSON |
**Return Type:** Returns a StructType with columns named `c0`, `c1`, `c2`,
etc., where each column corresponds to the extracted field values. All output
columns have nullable string type matching the input JSON expression's data
type.
**Supported Data Types:**
Input arguments must be string types with collation support (specifically
StringTypeWithCollation with trimming support). All children expressions must
accept string data types.
**Edge Cases:**
- Always returns at least one row (nullable = false), even for invalid JSON
- Null JSON input results in null values for all extracted fields
- Invalid or missing field paths return null for those specific columns
- Foldable (constant) field expressions are pre-evaluated for performance
optimization
- Requires minimum 2 arguments (JSON string + at least one field name),
throws QueryCompilationErrors.wrongNumArgsError otherwise
**Examples:**
```sql
-- Extract name and age from JSON
SELECT json_tuple('{"name":"John", "age":30, "city":"NYC"}', 'name', 'age')
AS (name, age);
-- Returns: John, 30
-- Extract nested fields
SELECT json_tuple('{"user":{"id":123,"email":"[email protected]"}}',
'user.id', 'user.email') AS (id, email);
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(json_tuple(col("json_col"), lit("field1"),
lit("field2")).as(Seq("c0", "c1")))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Large
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.JsonTuple`
**Related:**
- `get_json_object` - Extract single JSON field
- `from_json` - Parse JSON with explicit schema
- `json_extract` - Alternative JSON field extraction
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
