mazhengxuan created HBASE-30562:
-----------------------------------
Summary: Spark 4 connector reads the wrong Avro fields after
nested schema pruning
Key: HBASE-30562
URL: https://issues.apache.org/jira/browse/HBASE-30562
Project: HBase
Issue Type: Sub-task
Components: hbase-connectors
Reporter: mazhengxuan
Selecting a nested field from an Avro record in the Spark 4 HBase connector can
return the wrong value or fail with a `ClassCastException`.
`HBaseScan.readSchema()` reports the pruned struct, but `HBasePartitionReader`
decodes and returns the full struct from the catalog. Spark reads the field by
its position in the reported schema. If `second` is the only field in that
schema, Spark reads position 0, which still contains `first` in the returned
row.
### Minimal reproduction
# With Spark 4.0.0 and the Spark 4 HBase connector, create a table named
`nested_probe` in the `default` namespace with column family `cf`.
# Write one row with key `row1`. Store a single binary Avro record in `cf:p`
using the schema below, with `first = "first-value"`, `second =
"second-value"`, and `quantity = 42`.
# Read the table and run the two projections separately.
```scala
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
val avroSchema = """{
"type": "record",
"name": "Payload",
"fields": [
\{"name": "first", "type": "string"},
\{"name": "second", "type": "string"},
\{"name": "quantity", "type": "int"}
]
}"""
val catalog = """{
"table": \{"namespace": "default", "name": "nested_probe"},
"rowkey": "key",
"columns": {
"key": \{"cf": "rowkey", "col": "key", "type": "string"},
"payload": \{"cf": "cf", "col": "p", "avro": "payloadSchema"}
}
}"""
val df = spark.read
.format("org.apache.hadoop.hbase.spark.datasources.HBaseTableProvider")
.option("catalog", catalog)
.option("payloadSchema", avroSchema)
.option("hbase.spark.config.location", "/path/to/hbase-site.xml")
.load()
df.select("payload.second").collect()
df.select("payload.quantity").collect()
```
| Projection | Expected | Actual |
| --- | --- | --- |
| `payload.second` | `second-value` | `first-value` |
| `payload.quantity` | `42` | `ClassCastException`: `UTF8String` cannot be cast
to `Integer` |
Both queries return the expected values when
`spark.sql.optimizer.nestedSchemaPruning.enabled` is set to `false`. Reproduced
with Spark 4.0.0 and an HBase MiniCluster using connector commit `52e5231`.
### Proposed fix
Keep top-level column pruning in `HBaseScanBuilder.pruneColumns()`, but retain
the full catalog type for each selected column. This makes the scan schema
match what the reader returns and lets Spark select the nested fields. Add a
regression test for both projections with nested schema pruning enabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)