andygrove opened a new pull request, #5173: URL: https://github.com/apache/datafusion-comet/pull/5173
## Which issue does this PR close? No issue filed — mechanical compiler-warning cleanup, split out of #5141. Follows #5168 and #5170. ## Rationale for this change Scala 2.13 deprecates `Iterable.toIterable` (it is being made protected). Comet calls it at 12 sites, all of them `xs.toIterable.asJava` when handing a collection to a protobuf `addAll*` builder — **12 of the 127 warnings** under the 2.13 profiles. The interesting part is what *not* to do. At all 12 sites the receiver is an `Array`, so `.toIterable` resolves through `Predef.wrapRefArray` (the non-deprecated `Array` → `mutable.ArraySeq` wrap) and then `Iterable.toIterable` returns `this`. **The call is already free**, and the obvious fix of `.toSeq` would *add* a copy on 2.13, since `ArrayOps.toSeq` is `toIndexedSeq`. `.asJava` cannot be called on an `Array` directly either — that would need two chained implicit conversions. So rather than convert the arrays, this PR removes them. Every one of these `addAll*` calls then takes `.asJava` on a `Seq`, which is what the neighbouring `addAll*` calls in the same methods already do (e.g. `addAllFields(scanTypes.asJava)`, `addAllPartitionByList(partitionExprs.map(_.get).asJava)`). ## What changes are included in this PR? - **`schema2Proto`** takes and returns a `Seq[StructField]` / `Seq[SparkStructField]`. All five callers already hold a `StructType`, which *is* a `Seq[StructField]`, so they pass it straight in instead of unwrapping `.fields`; `CometNativeShuffleWriter` drops a `.toArray` on the way in. This clears 6 of the 12 warnings on its own (`CometNativeScan` ×3, `CometCsvNativeScanExec` ×2, and it is the same array that `addAllDataSchema` and friends were wrapping). - **`QueryPlanSerde.serializeDataType`** maps over the `StructType` instead of `s.fields` for the struct field names/datatypes/nullability. That also drops a `.toSeq` (a real copy) on `fieldDatatypes`. - **Projection vectors** are built from the `StructType` in both scan serdes, and in `CometNativeScan` from a `Range` rather than `Array.range` — same values, no array. - **`CometNativeScan` default values**: the `zipWithIndex`/`filter`/`map`/`unzip` chain runs on an iterator and materializes once, so the three intermediate arrays are gone. - **`CometWindowExec.winExprs`** keeps the `Seq` that `op.windowExpression.map` returns instead of `.toArray`-ing it; it is only used for `.length` and `.map`. Net effect: 12 deprecation warnings gone, several array copies removed from the per-plan serde path, and no conversion added anywhere. ## How are these changes tested? No new tests — no intended behavior change; the protobuf messages built are identical. - Warnings under the default profile (Spark 4.1 / Scala 2.13): **127 → 115**, all 12 of these gone and none added, verified by diffing the full sorted warning list before and after. - Scala 2.12 (`-Pspark-3.5`): warning list **byte-identical** (15 → 15). - `test-compile` passes on all five profiles: default, `-Pspark-3.4`, `-Pspark-3.5`, `-Pspark-3.5 -Pscala-2.13`, `-Pspark-4.0`. - `spotless:check` and `scalastyle:check` pass. - 333 tests pass across the suites covering every touched serde path: `CometExecSuite` and `CometWindowExecSuite` (window serde), `CometCsvNativeReadSuite` and `CometCsvExpressionSuite` (`CometCsvNativeScanExec`, including its projection vector and both schema protos), `CometNativeShuffleSuite` (`schema2Proto` via `addExpectedOutputSchema`), `ParquetReadV1Suite` and `CometNativeReaderSuite` (`CometNativeScan` schemas, projection vector and default values, plus `serializeDataType` on nested structs). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
