andygrove opened a new issue, #3161:
URL: https://github.com/apache/datafusion-comet/issues/3161
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `json_object_keys` function,
causing queries using this function to fall back to Spark's JVM execution
instead of running natively on DataFusion.
The `JsonObjectKeys` expression extracts the keys from a JSON object and
returns them as an array of strings. This expression is implemented as a
runtime-replaceable unary expression that delegates to the
`JsonExpressionUtils.jsonObjectKeys` method for actual evaluation.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
json_object_keys(json_string)
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(expr("json_object_keys(json_column)"))
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| json_string | StringType | A valid JSON object string from which to
extract keys |
**Return Type:** `ArrayType(StringType)` - Returns an array of strings
containing the keys from the JSON object.
**Supported Data Types:**
- String types with collation support (including trim collation)
- Input must be a valid JSON object string
**Edge Cases:**
- **Null handling**: Returns null when input is null (expression is nullable)
- **Empty JSON object**: Returns empty array for `{}`
- **Invalid JSON**: Behavior depends on underlying `JsonExpressionUtils`
implementation
- **Non-object JSON**: Arrays, primitives, and other JSON types may return
null or throw exceptions
- **Nested objects**: Only extracts top-level keys, does not traverse nested
structures
**Examples:**
```sql
-- Extract keys from a simple JSON object
SELECT json_object_keys('{"name": "John", "age": 30, "city": "NYC"}');
-- Result: ["name", "age", "city"]
-- Handle empty JSON object
SELECT json_object_keys('{}');
-- Result: []
-- Handle null input
SELECT json_object_keys(NULL);
-- Result: NULL
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq(
("""{"f1": "value1", "f2": "value2"}"""),
("""{"a": 1, "b": 2, "c": 3}"""),
(null)
)).toDF("json_col")
df.select(expr("json_object_keys(json_col)")).show()
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Large
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.JsonObjectKeys`
**Related:**
- `get_json_object` - Extract specific values from JSON
- `json_extract` - Extract JSON values using path expressions
- `from_json` - Parse JSON strings into structured data
- Other JSON manipulation functions in the `json_funcs` group
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
