[
https://issues.apache.org/jira/browse/BEAM-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kasia Kucharczyk updated BEAM-9896:
-----------------------------------
Description:
SnowflakeIO has currently only batch processing. To provide streaming
processing it is required to use
[SnowPipe|https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html]
which enables streaming write to Snowflake. Currently it is impossible to
provide streaming read.
[Snowflake ingest library|https://github.com/snowflakedb/snowflake-ingest-java]
for Java provides ingest methods which provides using SnowPipe and loading data
by authorising with key-pair and providing staged files (ex. on GCS). It also
provides method retrieving load history which is the only way to check if files
could be loaded properly. Checking history can be only provided by actively
requesting API because loading data is asynchronous.
was:
SnowflakeIO has currently only batch processing. To provide streaming
processing it is required to use
[SnowPipe|[https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html]]
which enables streaming write to Snowflake. Currently it is impossible to
provide streaming read.
[Snowflake ingest
library|[https://github.com/snowflakedb/snowflake-ingest-java]] for Java
provides ingest methods which provides using SnowPipe and loading data by
authorising with key-pair and providing staged files (ex. on GCS). It also
provides method retrieving load history which is the only way to check if files
could be loaded properly. Checking history can be only provided by actively
requesting API because loading data is asynchronous.
> Add streaming for SnowflakeIO.Write to Java SDK
> -----------------------------------------------
>
> Key: BEAM-9896
> URL: https://issues.apache.org/jira/browse/BEAM-9896
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
> Reporter: Dariusz Aniszewski
> Assignee: Kasia Kucharczyk
> Priority: P2
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> SnowflakeIO has currently only batch processing. To provide streaming
> processing it is required to use
> [SnowPipe|https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html]
> which enables streaming write to Snowflake. Currently it is impossible to
> provide streaming read.
> [Snowflake ingest
> library|https://github.com/snowflakedb/snowflake-ingest-java] for Java
> provides ingest methods which provides using SnowPipe and loading data by
> authorising with key-pair and providing staged files (ex. on GCS). It also
> provides method retrieving load history which is the only way to check if
> files could be loaded properly. Checking history can be only provided by
> actively requesting API because loading data is asynchronous.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
