Samiflux commented on PR #8951: URL: https://github.com/apache/hop/pull/8951#issuecomment-6438064476
Hi, thanks for this fix! I'm trying to get more involved in Apache Hop development whenever I can, although my free time is limited, so apologies for the delayed follow-up. I tested your minimal pipeline (Generate Rows → Generate Random Value, 1000 rows), and it runs successfully on Windows with Hop 2.20.0-SNAPSHOT, Java 21 and Spark 4.1.3. This is a good sign: the Windows fallback allows this minimal pipeline to run without setting `HADOOP_HOME` or `hadoop.home.dir`. However, my pipeline, which I shared earlier in this PR, uses Spark File Input → Sample Rows → Spark File Output to read a local CSV file and write the result to a local `file:///` path. It still fails with this error: ```text java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)' ``` Full log: ```text 2026/10/27 09:58:50 [Basic] test - Prepared native Spark pipeline engine with run configuration 'spark-local' (master=local[*]) 2026/10/27 09:58:50 [Basic] test - Reusing default SparkSession (version=4.1.3) 2026/10/27 09:58:51 [Error] test - Error executing Spark pipeline 2026/10/27 09:58:51 [Error] test - java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)' 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:817) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1415) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1620) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:802) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2078) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2122) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:1020) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2078) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2122) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404) 2026/10/27 09:58:51 [Error] test - at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:184) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$writeAndCommit$3(FileFormatWriter.scala:275) 2026/10/27 09:58:51 [Error] test - at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:496) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.writeAndCommit(FileFormatWriter.scala:275) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeWrite(FileFormatWriter.scala:306) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:189) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:117) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:115) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:129) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:185) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$8(SQLExecution.scala:177) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:285) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$7(SQLExecution.scala:139) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:106) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:139) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:308) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:138) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:92) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:250) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:185) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:717) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:184) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:201) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:194) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:491) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:491) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:37) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:360) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:356) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:467) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:194) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyCommandExecuted$1(QueryExecution.scala:155) 2026/10/27 09:58:51 [Error] test - at scala.util.Try$.apply(Try.scala:217) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1392) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.util.LazyTry.tryT$lzycompute(LazyTry.scala:46) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.util.LazyTry.tryT(LazyTry.scala:46) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.util.LazyTry.get(LazyTry.scala:58) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:160) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:239) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.classic.DataFrameWriter.runCommand(DataFrameWriter.scala:592) 2026/10/27 09:58:51 [Error] test - at org.apache.spark.sql.classic.DataFrameWriter.save(DataFrameWriter.scala:115) 2026/10/27 09:58:51 [Error] test - at org.apache.hop.spark.pipeline.handler.SparkFileIoSupport.writeDataset(SparkFileIoSupport.java:119) 2026/10/27 09:58:51 [Error] test - at org.apache.hop.spark.pipeline.handler.SparkFileOutputHandler.handleTransform(SparkFileOutputHandler.java:149) 2026/10/27 09:58:51 [Error] test - at org.apache.hop.spark.pipeline.HopPipelineMetaToSparkConverter.createDataset(HopPipelineMetaToSparkConverter.java:267) 2026/10/27 09:58:51 [Error] test - at org.apache.hop.spark.engines.SparkPipelineEngine.lambda$0(SparkPipelineEngine.java:344) 2026/10/27 09:58:51 [Error] test - at java.base/java.lang.Thread.run(Thread.java:1583) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
