ouwei-xhs opened a new issue, #43333:
URL: https://github.com/apache/arrow/issues/43333

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   I've written a program to convert my data to parquet format. during my test, 
I've encountered such an error on FileWrite::Close
   ```
   parquet_writer.cpp:186 Failed to close writer: IOError: Couldn't serialize 
thrift: Internal buffer size overflow
   ```
   the corresponding implementation is:
   <img width="751" alt="image" 
src="https://github.com/user-attachments/assets/a4c937d4-8385-4852-9a57-2b4e07eb9e84";>
   
   it seems like the case mentioned in this 
[issue](https://github.com/apache/arrow/issues/2077)
   
   during my test, I've also found that the error is related to the column 
count.
   if the schema contains 8 columns, then it fails:
   ```
   parquet_writer.cpp:172 The table constains: [8] columns and [70409] rows
   parquet_writer.cpp:186 Failed to close writer: IOError: Couldn't serialize 
thrift: Internal buffer size overflow
   ```
   
   but it contains less than 8, then it works well:
   ```
   parquet_writer.cpp:172 The table constains: [7] columns and [70409] rows
   processor.cpp:41 Try to upload parquet file 
/data/ssd0/redtable9/redzippydb/5/parquet-output/6/test_table_null/10.13.15.13-6-0.parquet,
 file size: 7072759
   ```
   
   Is there any params to tune to solve this issue?
   
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to