voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3841774713
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/TestSparkSqlHudiPackageStructure.java:
##########
@@ -89,6 +98,90 @@ public void
testSparkSqlHudiScalaTestClassesInAllowedPackagesOnly() {
"Expected to find at least one Scala test class in " + BASE_PACKAGE);
}
+ /**
+ * Every Scala test class under {@link #BASE_PACKAGE} must be named by at
least one Azure
+ * wildcardSuites entry, otherwise it silently never runs on Azure.
+ *
+ * <p>The Azure jobs deliberately name leaf packages ({@code dml.others},
{@code dml.insert},
+ * {@code dml.schema}) rather than the recursive {@code dml} parent, because
ScalaTest's
+ * {@code -w} is a plain prefix match with no exclusion primitive: pointing
one job at
+ * {@code ...hudi.dml} would re-run the whole {@code dml.insert} set that
already has its own
+ * job. That split is what makes a newly added {@code dml.*} package start
out dark, so this
+ * test is the guard for it - the other, non-recursive {@code
testSparkSqlHudi...} check above
+ * lets {@code dml.*} through because it treats {@code dml} as one allowed
package.
+ */
+ @Test
+ public void testScalaTestPackagesAreCoveredByAzureWildcardSuites() {
+ Set<String> azurePrefixes = readAzureWildcardSuitePrefixes();
+ assertFalse(azurePrefixes.isEmpty(),
+ "Expected to parse at least one wildcardSuites entry from " +
AZURE_PIPELINE_FILE);
+
+ List<String> uncovered = findScalaTestClasses().stream()
+ .filter(className -> azurePrefixes.stream()
+ .noneMatch(prefix -> className.equals(prefix) ||
className.startsWith(prefix + ".")))
Review Comment:
Correct on both counts, and worth doing now rather than as a follow-up,
since a guard that overstates what it checks is the thing this PR keeps getting
wrong.
Rewritten in bbbd39b9: the test resolves the pipeline instead of scanning it
- `parameters:` lists, the `variables:` entries that join them, then each step
carrying both `-DwildcardSuites` and `-pl`, with `$(VAR)` expanded through that
map. A class counts as run only when some step both builds its module and names
its package, so a parameter list no step expands now contributes nothing.
Renamed to `testScalaTestPackagesAreRunByAnAzureStep`.
Verified it still passes as-is and fails on each of the three holes: a new
`dml.darkpkg`, deleting the step that expands
`JOB6_SPARK_DDL_OTHERS_WILDCARD_SUITES` while keeping the list, and repointing
that step's `-pl` away from `hudi-spark`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
