peterxcli commented on PR #3369:
URL:
https://github.com/apache/datafusion-comet/pull/3369#issuecomment-3864062169
> Could you verify how this behaves when a map has a null key? For example:
```python
from pyspark.sql import SparkSession
comet_jar_path = "/Users/.../comet-spark-spark4.0_2.13-0.14.0-SNAPSHOT.jar"
# spark = SparkSession.builder.appName("test").getOrCreate()
spark = SparkSession.builder.appName("test")\
.config("spark.jars", comet_jar_path)\
.config("spark.driver.extraClassPath", comet_jar_path)\
.config("spark.executor.extraClassPath", comet_jar_path)\
.config("spark.plugins", "org.apache.spark.CometPlugin")\
.config("spark.shuffle.manager",
"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")\
.config("spark.comet.explainFallback.enabled", "true")\
.config("spark.comet.expression.allowIncompatible", "true")\
.getOrCreate()
query = "SELECT map_contains_key(map_from_arrays(array(1, NULL), array('a',
'b')), 5)"
spark.sql(query).show()
spark.sql(query).explain(True)
```
I tested the sql, and both the vanilla spark and comet native will show:
```sql
pyspark.errors.exceptions.captured.SparkRuntimeException: [NULL_MAP_KEY]
Cannot use null as map key. SQLSTATE: 2200E
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
