Copilot commented on code in PR #12278:
URL: https://github.com/apache/gluten/pull/12278#discussion_r3417746327
##########
cpp/velox/substrait/SubstraitToVeloxExpr.cc:
##########
@@ -217,13 +217,29 @@ std::shared_ptr<const core::FieldAccessTypedExpr>
SubstraitVeloxExprConverter::t
auto inputColumnType = inputType;
for (;;) {
auto idx = tmp->field();
- fieldAccess = makeFieldAccessExpr(inputColumnType->nameOf(idx),
inputColumnType->childAt(idx), fieldAccess);
+ VELOX_USER_CHECK(
+ idx >= 0 && static_cast<uint32_t>(idx) < inputColumnType->size(),
+ "Field reference index {} is out of range for the {}-field row
type.",
+ idx,
+ inputColumnType->size());
+ const TypePtr childType = inputColumnType->childAt(idx);
+ fieldAccess = makeFieldAccessExpr(inputColumnType->nameOf(idx),
childType, fieldAccess);
if (!tmp->has_child()) {
break;
}
- inputColumnType = asRowType(inputColumnType->childAt(idx));
+ // Descending into a nested field is only valid when the current child
is
+ // itself a struct/row. For array/map/primitive children (e.g. a field
+ // nested under an array, as in Delta's "updating array type" case)
+ // asRowType() returns null; previously the next loop iteration
+ // dereferenced that null RowType and crashed the process with a
SIGSEGV.
+ // Throw a user error instead so plan validation fails cleanly and the
+ // query falls back to vanilla execution.
+ inputColumnType = asRowType(childType);
+ VELOX_USER_CHECK_NOT_NULL(
+ inputColumnType,
+ "Nested field reference into a non-struct type (e.g. an array or
map element) is not supported.");
Review Comment:
The new user-facing error messages don’t include enough context to debug
which field/path failed (only the index, and the non-struct case has no
index/type at all). Include the field index and preferably the current row type
/ field name and the child type (e.g., `childType->toString()` / kind) in both
checks so failures are actionable when diagnosing plan translation issues.
##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala:
##########
@@ -48,7 +48,21 @@ trait DeltaSQLCommandTest extends SharedSparkSession {
.set("spark.default.parallelism", "1")
.set("spark.memory.offHeap.enabled", "true")
.set("spark.sql.shuffle.partitions", "5")
- .set("spark.memory.offHeap.size", "2g")
+ // Bound native memory so a runaway suite (e.g. DeletionVectorsSuite's
+ // "2B rows" read, whose Velox native grows to ~13G) hits a HARD per-task
+ // cap and throws a clean Velox OOM -- a deterministic, baselineable test
+ // failure -- instead of growing until the kernel OOM-kills the whole
fork
+ // (which wedges sbt into the chronic shard hang).
`memory.isolation=true`
+ // makes the off-heap pool a hard per-task cap (= offHeap /
executorCores);
+ // `overAcquiredMemoryRatio=0` drops Gluten's 30% over-acquire backup.
The
+ // pool is bumped 2g->4g so the per-task cap stays a reasonable ~1-2G
while
+ // the total managed off-heap stays <= 4g, keeping the fork well under
the
+ // ~16G runner. NOTE: this is a HARD cap, so other genuinely memory-heavy
+ // tests may also start throwing clean OOMs -- the net failure delta is
to
+ // be measured from the run, not assumed.
+ .set("spark.memory.offHeap.size", "4g")
+ .set("spark.gluten.memory.isolation", "true")
+ .set("spark.gluten.memory.overAcquiredMemoryRatio", "0")
Review Comment:
The comment asserts a “reasonable ~1–2G” per-task cap but the cap depends on
`executorCores` (and/or task CPU settings), which aren’t pinned here—so the
effective cap may vary across environments and make OOM behavior less
deterministic than intended. Either (mandatory) set a fixed cores/task-cpu
configuration for these tests to stabilize the cap, or (optional) soften/update
the comment to avoid implying a stable per-task limit without controlling the
divisor.
##########
cpp/velox/substrait/SubstraitToVeloxExpr.cc:
##########
@@ -217,13 +217,29 @@ std::shared_ptr<const core::FieldAccessTypedExpr>
SubstraitVeloxExprConverter::t
auto inputColumnType = inputType;
for (;;) {
auto idx = tmp->field();
- fieldAccess = makeFieldAccessExpr(inputColumnType->nameOf(idx),
inputColumnType->childAt(idx), fieldAccess);
+ VELOX_USER_CHECK(
+ idx >= 0 && static_cast<uint32_t>(idx) < inputColumnType->size(),
+ "Field reference index {} is out of range for the {}-field row
type.",
+ idx,
+ inputColumnType->size());
+ const TypePtr childType = inputColumnType->childAt(idx);
+ fieldAccess = makeFieldAccessExpr(inputColumnType->nameOf(idx),
childType, fieldAccess);
Review Comment:
The new user-facing error messages don’t include enough context to debug
which field/path failed (only the index, and the non-struct case has no
index/type at all). Include the field index and preferably the current row type
/ field name and the child type (e.g., `childType->toString()` / kind) in both
checks so failures are actionable when diagnosing plan translation issues.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for
blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on
one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern),
recursive=True))
Review Comment:
The `"**/target/**/*.xml"` glob is very broad and can traverse/collect many
non-report XML files under `target/`, increasing runtime on large
workspaces/clones. Consider narrowing the search to known report directories
(e.g. `**/target/test-reports/**/*.xml`, `**/target/surefire-reports/**/*.xml`,
etc.) while keeping the `<testsuite>` root-tag guard as a safety net.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,656 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P
profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '1'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of
enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline
honest)'
+ type: boolean
+ required: false
+ default: true
+ pull_request:
+ paths:
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
+ - 'gluten-delta/**'
+ - 'backends-velox/src-delta40/**/DeltaSQLCommandTest.scala'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
Review Comment:
Setting `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true` weakens the
workflow’s security posture by opting into deprecated/insecure Node runtimes
for actions. If this isn’t required by a specific pinned action version, remove
it; if it is required, scope it as narrowly as possible (e.g., only for the
step that needs it) and document the dependency that forces it.
##########
.github/workflows/util/delta-spark-ut/known-failures.txt:
##########
@@ -0,0 +1,968 @@
+# Known Delta-on-Gluten unit test failures.
+#
+# Baseline of delta-io/delta `spark` ScalaTest tests EXPECTED to fail under the
+# Gluten Velox bundle. The Delta Spark UT (Gluten) workflow enforces this list:
+# a failing test NOT listed here is a regression (fails CI); a listed test that
+# now passes should be removed. Format: <fully.qualified.SuiteName>#<test
name>.
+# Lines starting with '#' are comments. See README.md in this directory.
+#
+# ---------------------------------------------------------------------------
+# Full 16-shard baseline. Originally seeded from 15 of 16 shards (run
+# 27490052632); shard 2 then hung/OOM-crashed and was excluded. Shard 2's
+# 60 failures have now been merged in on top of that seed, so every shard
+# enforces clean. 954 known failures total.
Review Comment:
The hard-coded “954 known failures total” will become stale as the baseline
shrinks/grows, which can confuse readers and reviewers. Prefer removing the
fixed count (or regenerate it automatically as part of the aggregate job
header) so the file’s commentary stays accurate over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
