CurtHagenlocher commented on issue #3480: URL: https://github.com/apache/arrow-adbc/issues/3480#issuecomment-3366712365
The comments suggest that setting statement property `adbc.snowflake.statement.ingest_copy_concurrency` to 0 will cause ["only a single COPY query will be executed as part of ingestion once all files have finished uploading"](https://github.com/apache/arrow-adbc/blob/eaf4e07dcd509788430a7ebde0e27b319bae364c/go/adbc/driver/snowflake/bulk_ingestion.go#L97). This is not borne out either experimentally or by [code inspection](https://github.com/apache/arrow-adbc/blob/eaf4e07dcd509788430a7ebde0e27b319bae364c/go/adbc/driver/snowflake/bulk_ingestion.go#L550). As an alternative, I've modified the driver so that it only runs a single `COPY INTO` once all the uploads are done. I observed no performance difference, so the multiple and concurrent `COPY INTO`s are absolved of any blame here. I suspect I'll have to wait to hear back from Snowflake about the observed difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
