chenyuanxing opened a new issue, #34393:
URL: https://github.com/apache/arrow/issues/34393

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   We experimented with the same data and found that the performance of Arrow 
deserialization is only 1/3 of PB. Arrow cost 8s and PB just cost 3s. 
   
   test data set:(A total of 1000 lines, each line has 10 fields, and each 
field has 2000 data.)
   272M    Arrow_data
   496M    PB_data
   
   During the implementation process, Arrow uses flatbuf to serialize and 
deserialize, which should be faster in theory. but it is not. 
   ```
   Schema subSchema = MessageSerializer.deserializeSchema(new 
ReadChannel(Channels.newChannel(new ByteArrayInputStream(schemaBytes))));
   ArrowRecordBatch subRecordBatch = 
MessageSerializer.deserializeRecordBatch(new 
ReadChannel(Channels.newChannel(new ByteArrayInputStream(dataBytes))), 
ArrowUtils.getAllocator());
   
   VectorSchemaRoot subRoot = VectorSchemaRoot.create(subSchema, 
ArrowUtils.getAllocator());
   VectorLoader subLoader = new VectorLoader(subRoot);
   subLoader.load(subRecordBatch);
   subRecordBatch.close();
   ```
   
   I looked at the source code and found that The main time-consuming is 
divided into two parts:
   one is :
   
![image](https://user-images.githubusercontent.com/16252583/222102013-ef9204fb-04bb-4c26-b7a7-7eb2a24edd58.png)
   the other is:
    `org.apache.arrow.vector.VectorLoader#load`
     
   Is there something wrong with my way of using it, or is the serialization 
performance of the java version of Arrow just poor?
   
   
   ### Component(s)
   
   Java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to