hansva opened a new issue, #9594:
URL: https://github.com/apache/hop/issues/9594
### What would you like to happen?
The fat jar builder lives in the Beam engine plugin:
- Building a fat jar for the **native Spark engine** needs the Beam plugin
installed. In Hop GUI it also needs `HOP_SPARK_CLIENT_VERSION` set.
- The Spark engine can't build its own jar, so Databricks *Deploy & run* and
cluster runs need a jar built by hand first.
- A native Spark fat jar includes everything installed, including the Beam
SDK and the Spark and Hadoop jars the cluster already ships. Spark loads
classes parent-first, so it never uses those copies.
- A build takes about a minute and shows nothing in between. Both the log
and the GUI dialog look stuck.
**What changes**
*The builder moves to core* (`org.apache.hop.core.fatjar`):
- `FatJarBuilder` moves from the Beam plugin. It now uses buffered IO;
unbuffered 1 kB writes were ten times slower on slow file systems.
- New `FatJarCache` keeps one jar per engine and installation, with a
`.sha256` file next to it. It is rebuilt only when the installed jars change.
The folder is set with `HOP_FAT_JAR_CACHE_FOLDER` and defaults to the system
temporary folder.
- The jar selection for native Spark (`NativeSparkFatJar`) moves to core
too. The Beam plugin keeps its own selection in `BeamFatJar`, which the
generator calls through the plugin registry.
*Choosing the engine:*
- `hop-conf --generate-fat-jar` moves to the engine module. It gets
`--fat-jar-engine=spark|beam` and `--include-spark-runtime`. The old
`--spark-client-version=native[-provided]` still works.
- *Tools → Generate a Hop fat jar* moves to Hop GUI itself. When both
engines are installed, it asks which engine the jar is for.
- A native Spark jar leaves out the Beam engine plugin and the Beam SDK jars.
*A smaller native Spark jar:*
- The Spark engine plugin now ships its Maven dependency tree
(`spark-dependencies.txt`). The fat jar leaves out every jar that a Spark
distribution of that version ships (the Spark and Hadoop dependencies), except
the lakehouse connectors. `--include-spark-runtime` keeps them.
- Only the Linux x86_64 SWT jar is included. Plugin classes reference SWT
types, but the native libraries for the other platforms aren't needed.
- With Beam also installed, the jar goes from 776 to 595 MB, and the build
from 71 to 47 seconds.
*Progress:*
- The build logs a line every 10% of the jars ("Fat jar: 240 of 480 jar
files added (50%)").
- In Hop GUI, a progress dialog shows each jar as it is added and has a
working *Cancel*. A canceled build deletes the partial jar.
*Databricks:* *Deploy & run* uploads the cached jar when no fat jar is set.
*Docs:* the Spark pages no longer ask for the Beam plugin (the standalone
cluster guide keeps it for its sample data). The commands use
`--fat-jar-engine=spark`.
**Compatibility**
- Existing `hop-conf` commands keep working.
- The translated strings of the old Beam GUI dialog are removed, and the new
dialog has English strings only for now.
### Issue Priority
Priority: 2
### Issue Component
Component: Beam, Component: Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]