lwhite1 commented on PR #13481: URL: https://github.com/apache/arrow/pull/13481#issuecomment-1171450618
> @lwhite1 ignore the integration test. > > The important one is the JNI pipeline. > > ``` > 14:25:59.543 [main] DEBUG org.apache.arrow.memory.BaseAllocator - closed allocator[ROOT]. > Error: Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.185 s <<< FAILURE! - in org.apache.arrow.dataset.jni.TestReservationListener > Error: testCustomReservationListener Time elapsed: 0.105 s <<< ERROR! > java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so > at org.apache.arrow.dataset.jni.TestReservationListener.testCustomReservationListener(TestReservationListener.java:75) > Error: testDirectReservationListener Time elapsed: 0.078 s <<< ERROR! > java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so > at org.apache.arrow.dataset.jni.TestReservationListener.testDirectReservationListener(TestReservationListener.java:45) > [INFO] > [INFO] Results: > [INFO] > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > [INFO] > [INFO] > [INFO] Results: > [INFO] > Error: Failures: > Error: TestFileSystemDatasetFactory.testCloseAgain:41 Unexpected exception thrown: java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so > Error: TestFileSystemDatasetFactory.testErrorHandling:36 expected: <illegal file format id: -1> but was: <error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so> > Error: Errors: > Error: TestFileSystemDataset.testBaseParquetRead:77 » IllegalState error loading nati... > Error: TestFileSystemDataset.testEmptyProjectSelectsZeroColumns:170 » IllegalState er... > Error: TestFileSystemDataset.testErrorThrownWhenExecuteTaskAfterTaskClose:290 » IllegalState > Error: TestFileSystemDataset.testErrorThrownWhenIterateOnIteratorAfterTaskClose:306 » IllegalState > Error: TestFileSystemDataset.testErrorThrownWhenScanAfterScannerClose:276 » IllegalState > Error: TestFileSystemDataset.testErrorThrownWhenScanAgain:231 » IllegalState error lo... > Error: TestFileSystemDataset.testNoErrorWhenCloseAgain:215 » IllegalState error loadi... > Error: TestFileSystemDataset.testNullProjectSelectsAllColumns:193 » IllegalState erro... > Error: TestFileSystemDataset.testParquetBatchSize:129 » IllegalState error loading na... > Error: TestFileSystemDataset.testParquetDirectoryRead:153 » IllegalState error loadin... > Error: TestFileSystemDataset.testParquetProjectSingleColumn:100 » IllegalState error ... > Error: TestFileSystemDataset.testScanInOtherThread:258 » IllegalState error loading n... > Error: TestReservationListener.testCustomReservationListener:75 » IllegalState error ... > Error: TestReservationListener.testDirectReservationListener:45 » IllegalState error ... > [INFO] > Error: Tests run: 16, Failures: 2, Errors: 14, Skipped: 0 > [INFO] > [INFO] ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Arrow Orc Adapter 9.0.0-SNAPSHOT ................... SUCCESS [ 20.664 s] > [INFO] Arrow Gandiva ...................................... SKIPPED > [INFO] Arrow Java Dataset 9.0.0-SNAPSHOT .................. FAILURE [ 20.780 s] > [INFO] ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] ------------------------------------------------------------------------ > [INFO] Total time: 21.469 s (Wall Clock) > [INFO] Finished at: 2022-06-30T14:26:00Z > [INFO] ------------------------------------------------------------------------ > ``` > The important one is the JNI pipeline. oh that. IDK where that came from. When I built locally, it failed so I switched to master and did a clean build and it failed again. I assumed it was a local issue and the CI builds were unaffected. The issue is that the c-data maven module is expecting a variable with one name (for the build dir) and the other three cpp modules expect a different name for the same value. If you run the c-data-module on its own with the expected argument, it works fine. However, the build that creates the other JNI libs also calls the c-data module under the hood, and then it fails - I believe it's because it can't find the directory where the lib file should be. Locally, I got it to work in two ways: First, after building c-data separately, I deleted that step from the other JNI build and everything was fine as it used the previously compiled code. However, I was afraid to commit that change, thinking it would break the CI build. Instead I called the "other JNI" build passing the same value using both argument names. That also worked. I spent some time trying to figure out why this was happening. I couldn't see any related code changes since May so I thought it might just be my environment. We talked about the fact that the c-data-interface was built redundantly in the command line instructions, and whether one version should be removed, but it seemed risky to make that change. > After this reorganization, what is the expected process to build and test the JNI libraries? I didn't (consciously) change anything that would require process changes. I will look at the maven build again and see if I can't figure out how to pass the build dir to the c-data module using the name it expects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
