andygrove opened a new issue, #6452:
URL: https://github.com/apache/datafusion-comet/issues/6452
### Describe the bug
`CometInMemoryTableScanExec` prints every one of its constructor fields in
its plan string, and one of them is the `CachedRDDBuilder`, whose `toString`
holds the whole cached plan, physical and logical. So every plan that reads a
relation cached in Comet's format carries that plan inline in the scan's line,
with its raw newlines, which breaks the tree. It happens in `EXPLAIN` and in
the `Arguments:` line of `EXPLAIN FORMATTED`. The line also prints the
serializer's object identity (`ArrowCachedBatchSerializer@62857555`) and both
attribute lists.
Other Comet operators override `stringArgs` to leave out their
`originalPlan`. This one does not.
### Steps to reproduce
On `main` at 8e4bded55, with `spark.comet.exec.inMemoryCache.enabled=true`:
```scala
spark.range(0, 1000, 1, 2).selectExpr("id", "id % 10 AS
k").createOrReplaceTempView("named_t")
spark.catalog.cacheTable("named_t")
spark.sql("SELECT k, count(*) FROM named_t GROUP BY k").explain()
```
```
+- CometInMemoryTableScan Scan In-memory table named_t [k#1L],
org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer@62857555,
CachedRDDBuilder(org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer@62857555,StorageLevel(disk,
memory, deserialized, 1 replicas),*(1) Project [id#0L, (id#0L % 10) AS k#1L]
+- *(1) Range (0, 1000, step=1, splits=2)
,Some(named_t),SubqueryAlias named_t
+- View (`named_t`, [id#0L, k#1L])
+- Project [id#0L, (id#0L % cast(10 as bigint)) AS k#1L]
+- Range (0, 1000, step=1, splits=Some(2))
), [id#0L, k#1L], [k#1L]
```
### Expected behavior
The scan is described the way Spark's own cache scan is, for example
`CometInMemoryTableScan Scan In-memory table named_t [k#1L]`, followed by any
pruning predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
