Niivii commented on PR #2494: URL: https://github.com/apache/arrow-adbc/pull/2494#issuecomment-2668577412
> @davlee1972 Please feel free to reply here if you can give me more information (number of rows / columns / batches, etc) about your dataset that can help me to try to reproduce your failure (also, please try to build it fresh locally with the updated changes and see if that helps). In the meantime, I have some ideas about reducing the extraneous copy calls that have 0 rows inserted. From his photo, you can see the number of inserted rows in Snowflake on the right. This is roughly 5 times more than on my side but ofc it depends of the data type and number of columns too. I guess you might be able to replicate by tweaking my script and adding a 0 to the number of rows if you have the memory for it on your computer (should be around ~5Gb ?): `big_df = generate_big_dataframe(num_rows=50000000)` Otherwise, try to saturate your RAM with the biggest dataframe possible and see how it goes 🤷 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
