sermakarevich opened a new issue, #35840:
URL: https://github.com/apache/arrow/issues/35840

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Hi. We observe `orc.write_table` performance degradation compared to 
`parquet.write_table` when table has structs.  When we write same amount of 
data as a single flattened array - orc writer gives similar write time as 
parquet.  But we need to have these structs to be able to upload 
multidimensional arrays to Bigquery securing their original dimensionality.  
   
   ```
   pyarrow versions tested: 9.0.0, 11.0.0, 12.0.0
   os tested: MAC M1, Linux  
   ```
   
   Reproducible example:
   
   WRITING SRUCTS:
   ```
   import time
   import numpy as np
   import pyarrow as pa
   from pyarrow import orc, parquet, compute as pa_compute
   from matplotlib import pyplot as plt
   
   batch_sizes = [8, 16, 32, 64]
   
   orc_writetime, parquet_writetime = [], []
   
   for batch_size in batch_sizes:
   
     batch = np.random.rand(batch_size, 384, 400)
     array = pa.array(batch.reshape(-1))
   
     for dim in reversed(values.shape):
         array = pa.FixedSizeListArray.from_arrays(array, dim) 
         array = pa_compute.make_struct(array, field_names=["values"])
   
     table = pa.Table.from_arrays([array], names=["test"])
   
     start = time.time()
     with pa.BufferOutputStream() as buffer:
       orc.write_table(table, buffer,  compression="snappy")
     orc_writetime.append(time.time() - start)
   
     start = time.time()
     with pa.BufferOutputStream() as buffer:
       parquet.write_table(table, buffer,  compression="snappy")
     parquet_writetime.append(time.time() - start)
   
   plt.figure(figsize=(20,5))
   plt.plot(orc_writetime)
   plt.plot(parquet_writetime)
   plt.xticks(range(len(batch_sizes)), batch_sizes)
   plt.title("Writing structs experiment")
   plt.xlabel("batch_size")
   plt.ylabel("write time, seconds")
   plt.legend(["orc", "parquet"])
   plt.grid()
   ```
   
![image](https://github.com/apache/arrow/assets/9593371/18f2f2b7-51bb-40aa-9a5a-923b1231300e)
   
   
   WRITING ARRAYS
   ```
   orc_writetime, parquet_writetime = [], []
   
   for batch_size in batch_sizes:
   
     batch = np.random.rand(batch_size, 384, 400)
     array = pa.array(batch.reshape(-1))
   
     table = pa.Table.from_arrays([array], names=["test"])
   
     start = time.time()
     with pa.BufferOutputStream() as buffer:
       orc.write_table(table, buffer,  compression="snappy")
     orc_writetime.append(time.time() - start)
   
     start = time.time()
     with pa.BufferOutputStream() as buffer:
       orc.write_table(table, buffer,  compression="snappy")
     parquet_writetime.append(time.time() - start)
   
   
   plt.figure(figsize=(20,5))
   plt.plot(orc_writetime)
   plt.plot(parquet_writetime)
   plt.xticks(range(len(batch_sizes)), batch_sizes)
   plt.title("Writing arrays experiment")
   plt.xlabel("batch_size")
   plt.ylabel("write time, seconds")
   plt.legend(["orc", "parquet"])
   plt.grid()
   ```
   
![image](https://github.com/apache/arrow/assets/9593371/b7337dda-bcb6-473e-9a13-fb5eed87e908)
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to