soumilshah1995 opened a new issue, #14071:
URL: https://github.com/apache/iceberg/issues/14071
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When using try_variant_get (or variant_get) to extract fields from a VARIANT
column in Iceberg, filtering on the extracted field in the WHERE clause does
not return the expected results, even though the field can be successfully
selected and shows the correct type in the schema.
Create a users Iceberg table with a VARIANT column:
```
spark.sql("""
CREATE TABLE IF NOT EXISTS dev.default.users (
id BIGINT,
properties VARIANT
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
""")
```
Insert sample data:
```
from pyspark.sql.functions import col, parse_json
users_data = [
(1, '{ "name": "Alice", "age": 30, "address": {"city": "Wonderland"} }'),
(2, '{ "name": "Bob", "age": 40, "address": {"city": "Builderland"} }'),
(3, '{ "name": "Charlie", "age": 28, "address": {"city": "Dreamtown"} }')
]
df_users = spark.createDataFrame(users_data, ["id", "json_string"])
df_users_variant = df_users.withColumn("properties",
parse_json(col("json_string"))).drop("json_string")
df_users_variant.writeTo("dev.default.users").overwritePartitions()
```
```
df = spark.sql("""
SELECT
try_variant_get(properties, '$.name', 'string') AS name,
try_variant_get(properties, '$.age', 'int') AS age,
try_variant_get(properties, '$.address.city', 'string') AS city
FROM dev.default.users
""")
df.printSchema()
df.show()
root
|-- name: string (nullable = true)
|-- age: integer (nullable = true)
|-- city: string (nullable = true)
+-------+---+-----------+
| name|age| city|
+-------+---+-----------+
|Charlie| 28| Dreamtown|
| Alice| 30| Wonderland|
| Bob| 40|Builderland|
+-------+---+-----------+
```
Attempt to filter on the extracted field:
```
spark.sql("""
SELECT
try_variant_get(properties, '$.name', 'string') AS name,
try_variant_get(properties, '$.age', 'int') AS age,
try_variant_get(properties, '$.address.city', 'string') AS city
FROM dev.default.users
WHERE try_variant_get(properties, '$.age', 'int') >= 28
""").show(truncate=False)
+---+----+---+----+
|id |name|age|city|
+---+----+---+----+
+---+----+---+----+
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
