malinjawi opened a new issue, #13487:
URL: https://github.com/apache/gluten/issues/13487
### Backend
VL (Velox)
### Bug description
Reading a Lance table with a vector column fails when Gluten offloads the
Lance scan. With `spark.gluten.enabled=false`, the same read works.
```scala
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
import scala.collection.JavaConverters._
val path = "/tmp/vec.lance"
val vec = new MetadataBuilder().putLong("arrow.fixed-size-list.size",
4).build()
val schema = new StructType()
.add("id", IntegerType)
.add("vec", ArrayType(FloatType, containsNull = false), nullable = true,
vec)
val rows = Seq(Row(0, Seq(0f, 1f, 2f, 3f)), Row(1, Seq(4f, 5f, 6f, 7f)))
spark.createDataFrame(rows.asJava,
schema).write.format("lance").option("path", path).save()
spark.read.format("lance").option("path", path).load().collect()
// org.apache.spark.SparkUnsupportedOperationException:
// [UNSUPPORTED_ARROWTYPE] Unsupported arrow type FixedSizeList(4).
```
A table written directly with lance-core fails the same way, so this isn't
specific to lance-spark's writer.
The scan wraps every Arrow vector it gets from lance-spark in Spark's
`ArrowColumnVector`, and that class can't read `FixedSizeList`. Nothing stops
the offload, because `LanceScanTransformer.isExportable` only checks for a
pushed aggregation or a full-text query. Vanilla Spark works because
lance-spark reads through its own column vector class, which handles these
types.
I tried other column types and the metadata columns as well. All of these
get offloaded and then fail, while vanilla Spark returns the right rows:
- `FixedSizeBinary` and millisecond `Date` columns, and vectors inside a
struct, fail with the same `UNSUPPORTED_ARROWTYPE` error.
- JSON, blob (v1 and v2), unsigned integer and `LargeList` columns fail at
export. So do `_rowid`, `_rowaddr`, `_fragid`, `_row_created_at_version` and
`_row_last_updated_at_version`, and a scan that reads no columns, like `SELECT
1 FROM t`. lance-spark's export rejects them because the native schema doesn't
match the one Spark expects: "Arrow C stream export requires the native scan
schema to match the partition schema ...".
This goes back to #12958, which added the offload. Before #13280, the vector
case failed with `Unsupported data type: FixedSizeList(4)` instead. The code is
the same on main and branch-1.8, so 1.8 would ship with it.
It only happens while Gluten is active. With `spark.sql.ansi.enabled=true`,
Gluten falls back the whole plan, so Spark 4.0's default settings don't hit it.
With ANSI off, which Gluten needs on Spark 4, Spark 4.0.2 fails the same way as
3.5.5.
I'm working on a fix that makes these scans fall back at plan time. Unsigned
integer and `LargeList` columns can't be detected from the Spark schema:
lance-spark reads them as wider signed integers and plain lists, with no
marker. Those also need a small change in lance-spark.
Related: #12263
cc @sezruby @FelixYBW @zhouyuan
Written with help from AI (IBM Bob).
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
```
spark.plugins=org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=2g
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.sql.ansi.enabled=false
# only for the blob v2 and metadata-column cases
spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl=dir
spark.sql.catalog.lance.root=/tmp/lance-catalog
```
### System information
main at 381c806ec, local mode on macOS (arm64). Spark 3.5.5 (JDK 21) and
Spark 4.0.2 (JDK 17). lance-spark 0.8.0 and lance-core 12.0.0 from Maven
Central.
### Relevant logs
```bash
org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_ARROWTYPE]
Unsupported arrow type FixedSizeList(4).
at
org.apache.spark.sql.errors.ExecutionErrors.unsupportedArrowTypeError(ExecutionErrors.scala:143)
at
org.apache.spark.sql.errors.ExecutionErrors.unsupportedArrowTypeError$(ExecutionErrors.scala:140)
at
org.apache.spark.sql.errors.ExecutionErrors$.unsupportedArrowTypeError(ExecutionErrors.scala:218)
at
org.apache.spark.sql.util.ArrowUtils$.fromArrowType(ArrowUtils.scala:88)
at
org.apache.spark.sql.util.ArrowUtils$.fromArrowField(ArrowUtils.scala:147)
at org.apache.spark.sql.util.ArrowUtils.fromArrowField(ArrowUtils.scala)
at
org.apache.spark.sql.vectorized.ArrowColumnVector.<init>(ArrowColumnVector.java:141)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
