gortiz opened a new pull request, #19442:
URL: https://github.com/apache/pinot/pull/19442
## Summary
`SparkType.SPARK_3` declares the plugin name
`pinot-batch-ingestion-spark-3.2`, but the module has always been
`pinot-batch-ingestion-spark-3`.
`shouldLoadPlugin()` skips any plugin directory whose name contains `spark`,
so that only the ingestion plugin for the *selected* Spark version is
re-included — by comparing the directory name against
`SparkType#getPluginName()`:
```java
shouldLoadPlugin = shouldLoadPlugin ||
_sparkVersion.getPluginName().contentEquals(parentDir);
```
Since the declared name never matches a real directory, that clause is dead
and the ingestion plugin is never shipped to the executors. A Spark ingestion
job then fails with:
```
Caused by: java.lang.ClassNotFoundException:
org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentGenerationJobRunner
at
org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:463)
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142)
```
Passing `-pluginsToLoad pinot-batch-ingestion-spark-3` explicitly is
currently the only way to run one.
## Reproduction
Built a binary distribution from master and ran, with `SPARK_HOME` pointing
at a Spark 3.5.9 `scala2.13` build (Pinot depends on `spark-launcher_2.13`):
```
bin/pinot-admin.sh LaunchSparkDataIngestionJob -jobSpecFile <spec>
-pinotBaseDir <dist>
```
The job fails with the `ClassNotFoundException` above and produces no
segments. Adding `-pluginsToLoad pinot-batch-ingestion-spark-3` gets past it.
Note that segment generation still cannot complete on a Java 25 build for an
unrelated, already-known reason: Hadoop 3.4's `UserGroupInformation` calls
`Subject.getSubject`, which throws on JDK 23+, and
`-Djava.security.manager=allow` is rejected on JDK 25. This PR fixes the plugin
selection, which is as far as the job gets before that.
## Test plan
- [x] `LaunchSparkDataIngestionJobCommandTest` asserts every `SparkType`'s
plugin name resolves to a real module under
`pinot-plugins/pinot-batch-ingestion`, so a rename on either side is caught
instead of silently disabling the plugin again.
- [x] Verified the test fails without the fix: `SPARK_3 declares plugin name
'pinot-batch-ingestion-spark-3.2', which is not a module under
.../pinot-batch-ingestion`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
