bamaer commented on code in PR #9592:
URL: https://github.com/apache/hop/pull/9592#discussion_r4643436745


##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/engines/SparkPipelineEngine.java:
##########
@@ -291,11 +305,13 @@ public void prepareExecution() throws HopException {
 
   @Override
   public void startThreads() throws HopException {
+    HopVfsNamespace previousNamespace = null;
     ClassLoader pluginCl = getClass().getClassLoader();
     ClassLoader previousCl = Thread.currentThread().getContextClassLoader();
     try {
       // Spark 4 resolves classic.SparkSession via the context classloader / 
plugin CL
       Thread.currentThread().setContextClassLoader(pluginCl);
+      previousNamespace = HopVfsNamespaces.bindThread(vfsNamespace);

Review Comment:
   **Blocker:** this only works for the first Spark run in the JVM. The Spark 
task threads get the namespace by inheriting it from the thread that creates 
the SparkContext. The next run reuses that session, because nothing calls 
`cleanup()` on Hop Server or hop-run. Its tasks still carry the first run's 
namespace, which is closed by then, and the path goes back to a local folder.
   
   Repro: `aHopTransformOnSparkWritesThroughTheRunsNamedConnection` run twice 
in one JVM, a new `MemoryMetadataProvider` each time, no `cleanup()`. Run 1 
gives errors=0. Run 2 logs `Reusing active SparkSession` and gives errors=1 
with `Parent directory file:///…/vfs9591: does not exist!`.
   
   Suggested fix: bind the mini-pipeline's own namespace on the task thread in 
`HopMapPartitionsFn`. That pipeline already takes a namespace for the 
serialized metadata. Text File Output just doesn't find it through its 
variables.
   
   ```java
   // in call(), after `rows` is built
   rows.vfsNamespace = HopVfsNamespaces.resolve(pipeline);
   
   // PartitionIterator
   HopVfsNamespace vfsNamespace;
   
   @Override
   public boolean hasNext() {
     HopVfsNamespace previous = HopVfsNamespaces.bindThread(vfsNamespace);
     try {
       return hasNextBound(); // the current body of hasNext()
     } finally {
       HopVfsNamespaces.restoreThread(previous);
     }
   }
   ```
   
   With this, both runs pass, and the PR's test passes even without the engine 
change. It should also cover executors in other JVMs, which never see the 
driver's binding (not tested on a cluster). A second run in the test would keep 
this covered.



##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/engines/SparkPipelineEngine.java:
##########
@@ -430,6 +448,7 @@ public void run() {
       throw new HopException("Unexpected error starting Spark pipeline", e);

Review Comment:
   **Should fix:** if `startThreads()` fails before the `hop-spark-pipeline` 
thread starts, the namespace taken in `prepareExecution()` is never released. A 
wrong master or an unreachable cluster in `createSparkSession()` is enough. 
Only that thread and `cleanup()` release it, and Hop Server and hop-run don't 
call `cleanup()`. On a server that leaks one file system manager per failed 
start. Calling `releaseVfsNamespace()` in this catch fixes it. It already 
guards against a second release.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to