[
https://issues.apache.org/jira/browse/BEAM-12715?focusedWorklogId=633668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633668
]
ASF GitHub Bot logged work on BEAM-12715:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Aug/21 16:00
Start Date: 04/Aug/21 16:00
Worklog Time Spent: 10m
Work Description: aromanenko-dev merged pull request #15255:
URL: https://github.com/apache/beam/pull/15255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 633668)
Time Spent: 50m (was: 40m)
> SnowflakeWrite fails in batch mode when the number of shards is > 1000
> ----------------------------------------------------------------------
>
> Key: BEAM-12715
> URL: https://issues.apache.org/jira/browse/BEAM-12715
> Project: Beam
> Issue Type: Bug
> Components: io-java-snowflake
> Reporter: Daniel Mateus Pires
> Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When writing to Snowflake in batch mode, if the number of files to import is
> more than 1000, the load will fail
> From the Snowflake docs
> {quote}Of the three options for identifying/specifying data files to load
> from a stage, providing a discrete list of files is generally the fastest;
> however, the FILES parameter supports a maximum of 1,000 files, meaning a
> COPY command executed with the FILES parameter can only load up to 1,000
> files.
> {quote}
> I noticed that the Snowflake Write in batch mode ignores the number of shards
> set by the user, and I think the first step should be to get the number of
> shards before writing.
> Longer term, should Beam issue multiple COPY statements with a distinct list
> of files when the number of files is more than 1000? Maybe inside the same
> transaction (BEGIN; END; block)
>
> Also, I wanted to set the Jira issue component as io-java-snowflake but it
> does not exist
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
