sermakarevich opened a new issue, #35840:
URL: https://github.com/apache/arrow/issues/35840
### Describe the bug, including details regarding any error messages,
version, and platform.
Hi. We observe `orc.write_table` performance degradation compared to
`parquet.write_table` when table has structs. When we write same amount of
data as a single flattened array - orc writer gives similar write time as
parquet. But we need to have these structs to be able to upload
multidimensional arrays to Bigquery securing their original dimensionality.
```
pyarrow versions tested: 9.0.0, 11.0.0, 12.0.0
os tested: MAC M1, Linux
```
Reproducible example:
WRITING SRUCTS:
```
import time
import numpy as np
import pyarrow as pa
from pyarrow import orc, parquet, compute as pa_compute
from matplotlib import pyplot as plt
batch_sizes = [8, 16, 32, 64]
orc_writetime, parquet_writetime = [], []
for batch_size in batch_sizes:
batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))
for dim in reversed(values.shape):
array = pa.FixedSizeListArray.from_arrays(array, dim)
array = pa_compute.make_struct(array, field_names=["values"])
table = pa.Table.from_arrays([array], names=["test"])
start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)
start = time.time()
with pa.BufferOutputStream() as buffer:
parquet.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)
plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing structs experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```

WRITING ARRAYS
```
orc_writetime, parquet_writetime = [], []
for batch_size in batch_sizes:
batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))
table = pa.Table.from_arrays([array], names=["test"])
start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)
start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)
plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing arrays experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```

### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]