luoyuxia commented on PR #904: URL: https://github.com/apache/arrow-java/pull/904#issuecomment-3545626074
> > > > @V-Fenil Hi, thanks for your interest. I already built .so(for linux), .dylib(for mac). But I don't have windows env, so I can't provide .dll for you. To verify this pr, you'll need to build from source, see https://github.com/apache/arrow-java?tab=readme-ov-file#building-from-source. That's also what I did to verify my pr. > > > > > > > > > Hi @luoyuxia I'm testing your PR on linux. Could you share the built libarrow_dataset_jni.so file? I can build java but need the native library. (more specific my build was success but I can't find .so file) > > > Total build time was 49 mins And Arrow Java C Data Interface & Arrow Java Dataset was only 45 sec each!! So there was no C++ compilation I guess, if would be better if you share direct file. > > > > > > Of course I can share it. I can share you the `libarrow_dataset_jni.so` as well as the jar built with `libarrow_dataset_jni.so`. How can I share it? Send it to you email or by other way? > > I'm testing PR #904 (native Parquet writer via JNI) on Ubuntu 22.04/WSL2 with Java 11 and hitting a consistent failure during ParquetWriter initialization. > > Setup: > > * Downloaded jni-linux-x86_64 artifacts from CI build (run #19222857860) > * Using Arrow Java 19.0.0-SNAPSHOT with both libarrow_dataset_jni.so and > libarrow_cdata_jni.so loaded > * All library dependencies resolved (ldd shows no missing libraries) > > Error: The ParquetWriter constructor fails at line 71 with a memory leak error during cleanup: > > java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (128) Allocator(ROOT) 0/128/4998/9223372036854775807 (res/actual/peak/limit) at org.apache.arrow.dataset.file.ParquetWriter.close(ParquetWriter.java:158) at org.apache.arrow.dataset.file.ParquetWriter.(ParquetWriter.java:71) > > Analysis: Looking at the bytecode, the constructor creates a RootAllocator, then calls either `ArrowSchema.allocateNew()` (line 24) or `Data.exportSchema()` (line 37), which throws an exception. The constructor's cleanup calls close(), which detects the 128-byte leak from the allocator created at line 14. > > Questions: > > 1. Are there additional native libraries or system dependencies required beyond > libarrow_dataset_jni.so and libarrow_cdata_jni.so? > 2. Is the CI build fully functional, or does it require Arrow C++ runtime libraries > to be installed separately? > 3. What's the expected initialization sequence for ParquetWriter with these JNI > libraries? > > The Java code is simply: java FileOutputStream fos = new FileOutputStream(outputPath); ParquetWriter writer = new ParquetWriter(fos, schema); > > Any guidance would be appreciated. Thanks for this PR - looking forward to using the native performance! Hi, what's your code? I used the following in my local mac os but can't reproduce it ``` ublic class ParquetWriteTest { @TempDir public Path tempDir; @Test void test() throws Exception{ String parquetFilePath = Paths.get("testParquetWriter.parquet").toString(); List<Field> fields = Arrays.asList( Field.nullable("id", new ArrowType.Int(32, true)), Field.nullable("name", new ArrowType.Utf8())); Schema arrowSchema = new Schema(fields); int[] ids = new int[] {1, 2, 3, 4, 5}; String[] names = new String[] {"Alice", "Bob", "Charlie", "Diana", "Errrrve"}; // Write Parquet file try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); FileOutputStream fos = new FileOutputStream(parquetFilePath); ParquetWriter writer = new ParquetWriter(fos, arrowSchema); VectorSchemaRoot vectorSchemaRoot = createData(allocator, arrowSchema, ids, names)) { writer.write(vectorSchemaRoot); } } private static VectorSchemaRoot createData( BufferAllocator allocator, Schema schema, int[] ids, String[] names) { // Create VectorSchemaRoot from schema VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator); // Allocate space for vectors (we'll write 5 rows) root.allocateNew(); // Get field vectors IntVector idVector = (IntVector) root.getVector("id"); VarCharVector nameVector = (VarCharVector) root.getVector("name"); // Write data to vectors for (int i = 0; i < ids.length; i++) { idVector.setSafe(i, ids[i]); nameVector.setSafe(i, names[i].getBytes()); } // Set the row count root.setRowCount(ids.length); return root; } } ``` I'll try to find time to reproduce it in linux. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
