[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=450324&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-450324 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 24/Jun/20 08:34 Start Date: 24/Jun/20 08:34 Worklog Time Spent: 10m Work Description: kkucharc merged pull request #11794: URL: https://github.com/apache/beam/pull/11794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 450324) Time Spent: 8h 10m (was: 8h) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Priority: P2 > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=450321&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-450321 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 24/Jun/20 08:30 Start Date: 24/Jun/20 08:30 Worklog Time Spent: 10m Work Description: kkucharc commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-648678063 Thank you all a lot for your work and effort :) All tests passed - I'm merging this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 450321) Time Spent: 8h (was: 7h 50m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Priority: P2 > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=450277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-450277 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 24/Jun/20 07:20 Start Date: 24/Jun/20 07:20 Worklog Time Spent: 10m Work Description: kkucharc commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-648642561 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 450277) Time Spent: 7h 50m (was: 7h 40m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Priority: P2 > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=449685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-449685
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 23/Jun/20 08:34
Start Date: 23/Jun/20 08:34
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r444055571
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +513,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getStorageIntegrationName();
+
+@Nullable
+abstract String getStagingBucketName();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setStorageIntegrationName(String
storageIntegrationName);
+
+ abstract Builder setStagingBucketName(String stagingBucketName);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write withTable(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * Name of the cloud bucket (GCS by now) to use as tmp location of CSVs
during COPY statement.
+ *
+ * @param stagingBucketName - String with the name of the bucket.
+ */
+public Write withStagingBucketName(String stagingBucketName) {
+ return toBuilder().setStagingBucketName(stagingBucketName).build();
+}
+
+/**
+ * Name of the Storage Integration in Snowflake to be used. See
+ *
https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html
for
+ * reference.
+ *
+ * @param integrationName - String with the name of the Storage
Integration.
+ */
+public Write withStorageIntegrationName(String integrationName) {
+ return toBuilder().setStorageIntegrationName(integrationName).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=449684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-449684 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 23/Jun/20 08:33 Start Date: 23/Jun/20 08:33 Worklog Time Spent: 10m Work Description: purbanow commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-647995341 @RyanSkraba Thanks a lot for your CR. We're going to add your CSV concern to our feature improvement list. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 449684) Time Spent: 7.5h (was: 7h 20m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Priority: P2 > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=449675&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-449675
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 23/Jun/20 07:52
Start Date: 23/Jun/20 07:52
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r444031088
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +513,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getStorageIntegrationName();
+
+@Nullable
+abstract String getStagingBucketName();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setStorageIntegrationName(String
storageIntegrationName);
+
+ abstract Builder setStagingBucketName(String stagingBucketName);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write withTable(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * Name of the cloud bucket (GCS by now) to use as tmp location of CSVs
during COPY statement.
+ *
+ * @param stagingBucketName - String with the name of the bucket.
+ */
+public Write withStagingBucketName(String stagingBucketName) {
+ return toBuilder().setStagingBucketName(stagingBucketName).build();
+}
+
+/**
+ * Name of the Storage Integration in Snowflake to be used. See
+ *
https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html
for
+ * reference.
+ *
+ * @param integrationName - String with the name of the Storage
Integration.
+ */
+public Write withStorageIntegrationName(String integrationName) {
+ return toBuilder().setStorageIntegrationName(integrationName).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=449397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-449397
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 22/Jun/20 16:59
Start Date: 22/Jun/20 16:59
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r443687101
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +513,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getStorageIntegrationName();
+
+@Nullable
+abstract String getStagingBucketName();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setStorageIntegrationName(String
storageIntegrationName);
+
+ abstract Builder setStagingBucketName(String stagingBucketName);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write withTable(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * Name of the cloud bucket (GCS by now) to use as tmp location of CSVs
during COPY statement.
+ *
+ * @param stagingBucketName - String with the name of the bucket.
+ */
+public Write withStagingBucketName(String stagingBucketName) {
+ return toBuilder().setStagingBucketName(stagingBucketName).build();
+}
+
+/**
+ * Name of the Storage Integration in Snowflake to be used. See
+ *
https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html
for
+ * reference.
+ *
+ * @param integrationName - String with the name of the Storage
Integration.
+ */
+public Write withStorageIntegrationName(String integrationName) {
+ return toBuilder().setStorageIntegrationName(integrationName).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition -
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=443867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-443867 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 10/Jun/20 17:26 Start Date: 10/Jun/20 17:26 Worklog Time Spent: 10m Work Description: purbanow edited a comment on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-642150480 Hey @RyanSkraba , We answered for all your comments. Will you find a moment to check again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 443867) Time Spent: 7h (was: 6h 50m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=443864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-443864 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 10/Jun/20 17:25 Start Date: 10/Jun/20 17:25 Worklog Time Spent: 10m Work Description: purbanow commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-642150480 Hey @RyanSkraba , will you find a moment to check again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 443864) Time Spent: 6h 40m (was: 6.5h) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=443862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-443862
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 10/Jun/20 17:24
Start Date: 10/Jun/20 17:24
Worklog Time Spent: 10m
Work Description: kkucharc commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r438289586
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=443866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-443866 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 10/Jun/20 17:26 Start Date: 10/Jun/20 17:26 Worklog Time Spent: 10m Work Description: purbanow edited a comment on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-642150480 Hey @RyanSkraba , We answered fo all your comments. Will you find a moment to check again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 443866) Time Spent: 6h 50m (was: 6h 40m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=440021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440021
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 09:22
Start Date: 02/Jun/20 09:22
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433739364
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.io.FileIO.Sink} for writing
CSV. */
+public class CSVSink implements FileIO.Sink {
Review comment:
In one of the PR's , we're planing to add `create dispositions` for
tables which is connected more with schema concept. Let's talk about it then
:)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440021)
Time Spent: 6h 20m (was: 6h 10m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439931
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 06:42
Start Date: 02/Jun/20 06:42
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433652793
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.io.FileIO.Sink} for writing
CSV. */
+public class CSVSink implements FileIO.Sink {
Review comment:
Thanks for spotting this. I removed `CSVSink` class and start using
`.via(TextIO.sink())` .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 439931)
Time Spent: 6h 10m (was: 6h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439895&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439895
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 05:27
Start Date: 02/Jun/20 05:27
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433627687
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439891&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439891
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 04:58
Start Date: 02/Jun/20 04:58
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433620166
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439884
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 04:15
Start Date: 02/Jun/20 04:15
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433610313
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439691 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 01/Jun/20 19:31 Start Date: 01/Jun/20 19:31 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-637058818 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439691) Time Spent: 5.5h (was: 5h 20m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439444
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 01/Jun/20 08:47
Start Date: 01/Jun/20 08:47
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433114311
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439346
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 01/Jun/20 05:42
Start Date: 01/Jun/20 05:42
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433050696
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438992&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438992 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 30/May/20 00:13 Start Date: 30/May/20 00:13 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-636243814 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 438992) Time Spent: 5h (was: 4h 50m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438719
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:46
Start Date: 29/May/20 12:46
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432458477
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438715&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438715
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:43
Start Date: 29/May/20 12:43
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432456811
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -339,10 +360,31 @@
emptyCollection
.apply(Wait.on(output))
.apply(ParDo.of(new CleanTmpFilesFromGcsFn(stagingBucketDir)));
-
return output;
}
+private void checkArguments(Location loc) {
+ // Either table or query is required. If query is present, it's being
used, table is used
+ // otherwise
+ checkArgument(loc != null, "via() is required");
+ checkArgument(
+ loc.getStorageIntegrationName() != null,
Review comment:
1. I think it is more readable with separate method but i agree with you
that `is odd -- the loc configuration is passed as a parameter while all the
rest of the configurations are accessed as methods.` that's why I changed
`checkArguments` method to stop accepting location as argument.
2. `SnowflakeIO.Read ` requires `storageIntegrationName ` to be always set
but `SnowflakeIO.Write ` doesn't require that. The check against null in
`SnowflakeServiceImpl` is corresponding to `SnowflakeIO.Write ` method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438715)
Time Spent: 4h 40m (was: 4.5h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438711
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:36
Start Date: 29/May/20 12:36
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432453524
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Yes, you're right. Thanks for spotting this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438711)
Time Spent: 4.5h (was: 4h 20m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438699
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:08
Start Date: 29/May/20 12:08
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432429729
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438698&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438698
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:05
Start Date: 29/May/20 12:05
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432427386
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Both `storageIntegrationName ` and `stagingBucketName` are always
required for the connection between Snowflake and GCP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438698)
Time Spent: 4h 10m (was: 4h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438697&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438697
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:03
Start Date: 29/May/20 12:03
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432437111
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -339,10 +360,31 @@
emptyCollection
.apply(Wait.on(output))
.apply(ParDo.of(new CleanTmpFilesFromGcsFn(stagingBucketDir)));
-
return output;
}
+private void checkArguments(Location loc) {
+ // Either table or query is required. If query is present, it's being
used, table is used
+ // otherwise
+ checkArgument(loc != null, "via() is required");
+ checkArgument(
+ loc.getStorageIntegrationName() != null,
Review comment:
I think keeping this part of code in separate method makes code more
readable.
I agree with you about that `is odd -- the loc configuration is passed as a
parameter while all the rest of the configurations are accessed as methods`
I changed `checkArguments` method to stop accepting location as argument.
WDYT?
Yes , `storageIntegrationName` must not be null . Thanks for spotting this
:)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438697)
Time Spent: 4h (was: 3h 50m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438696
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 12:02
Start Date: 29/May/20 12:02
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432437111
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -339,10 +360,31 @@
emptyCollection
.apply(Wait.on(output))
.apply(ParDo.of(new CleanTmpFilesFromGcsFn(stagingBucketDir)));
-
return output;
}
+private void checkArguments(Location loc) {
+ // Either table or query is required. If query is present, it's being
used, table is used
+ // otherwise
+ checkArgument(loc != null, "via() is required");
+ checkArgument(
+ loc.getStorageIntegrationName() != null,
Review comment:
I think keeping this part of code in separate method makes code more
readable.
I agree with you about that `is odd -- the loc configuration is passed as a
parameter while all the rest of the configurations are accessed as methods`
I changed `checkArguments` method to stop accepting location as argument.
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438696)
Time Spent: 3h 50m (was: 3h 40m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438692&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438692
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:51
Start Date: 29/May/20 11:51
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432432306
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438690
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:46
Start Date: 29/May/20 11:46
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432427386
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Both `storageIntegrationName ` and `stagingBucketName` are always
required for the connection between Snowflake and GCP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438690)
Time Spent: 3.5h (was: 3h 20m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438689&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438689
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:45
Start Date: 29/May/20 11:45
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432429729
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PCol
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438687
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:44
Start Date: 29/May/20 11:44
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432427386
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Both `storageIntegrationName ` and `stagingBucketName` are always
required for connection between Snowflake and GCP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438687)
Time Spent: 3h 10m (was: 3h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438686&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438686
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:43
Start Date: 29/May/20 11:43
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432427386
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Both `storageIntegrationName ` and `stagingBucketName` are always
required for proper connection between Snowflake and GCP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438686)
Time Spent: 3h (was: 2h 50m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438685
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:39
Start Date: 29/May/20 11:39
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432427386
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
Both `storageIntegrationName ` and `stagingBucketName` are always
required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438685)
Time Spent: 2h 50m (was: 2h 40m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438684
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:37
Start Date: 29/May/20 11:37
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432412298
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
It makes sense. I added `@Nullable `. Thanks a lot :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438684)
Time Spent: 2h 40m (was: 2.5h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438683
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:31
Start Date: 29/May/20 11:31
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432423695
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -257,23 +301,12 @@
}
/**
- * Name of the cloud bucket (GCS by now) to use as tmp location of CSVs
during COPY statement.
- *
- * @param stagingBucketName - String with the name of the bucket.
- */
-public Read withStagingBucketName(String stagingBucketName) {
- return toBuilder().setStagingBucketName(stagingBucketName).build();
-}
-
-/**
- * Name of the Storage Integration in Snowflake to be used. See
- *
https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html
for
- * reference.
+ * A location object which contains connection config between Snowflake
and GCP.
*
- * @param integrationName - String with the name of the Storage
Integration.
+ * @param location - an instance of {@link Location}.
*/
-public Read withIntegrationName(String integrationName) {
- return toBuilder().setIntegrationName(integrationName).build();
+public Read via(Location location) {
Review comment:
No reason. I changed to `withLocation ` to be more consistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438683)
Time Spent: 2.5h (was: 2h 20m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438670&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438670
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:07
Start Date: 29/May/20 11:07
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432413521
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
+ private String stagingBucketName;
+
+ public static Location of(SnowflakePipelineOptions options) {
+return new Location(options.getStorageIntegrationName(),
options.getStagingBucketName());
+ }
+
+ public static Location of(String storageIntegrationName, String
stagingBucketName) {
Review comment:
I added removed link
https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html
. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438670)
Time Spent: 2h 20m (was: 2h 10m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438669
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 11:04
Start Date: 29/May/20 11:04
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432412298
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
Review comment:
It makes sense. I added `@Nullable `. Thanks a lot :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438669)
Time Spent: 2h 10m (was: 2h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438667
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 29/May/20 10:58
Start Date: 29/May/20 10:58
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r432409517
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakePipelineOptions.java
##
@@ -111,23 +111,13 @@
void setLoginTimeout(String loginTimeout);
- @Description("External location name to connect to.")
- String getExternalLocation();
-
- void setExternalLocation(String externalLocation);
-
- @Description("Temporary GCS bucket name")
+ @Description("Temporary GCS bucket name.")
String getStagingBucketName();
void setStagingBucketName(String stagingBucketName);
- @Description("Storage integration - required in case the external stage is
not specified.")
- String getStorageIntegration();
-
- void setStorageIntegration(String integration);
-
- @Description("Stage name. Optional.")
- String getStage();
+ @Description("Storage integration name")
+ String getStorageIntegrationName();
Review comment:
Yes. I'm totally aware of that. --stage option is not being used
anywhere in the codebase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 438667)
Time Spent: 2h (was: 1h 50m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Assignee: Pawel Urbanowicz
>Priority: P2
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438665&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438665 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 29/May/20 10:54 Start Date: 29/May/20 10:54 Worklog Time Spent: 10m Work Description: purbanow commented on a change in pull request #11794: URL: https://github.com/apache/beam/pull/11794#discussion_r432407862 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/enums/package-info.java ## @@ -0,0 +1,2 @@ +/** Snowflake IO data types. */ Review comment: Thanks for spotting this 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 438665) Time Spent: 1h 50m (was: 1h 40m) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438663 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 29/May/20 10:54 Start Date: 29/May/20 10:54 Worklog Time Spent: 10m Work Description: purbanow commented on a change in pull request #11794: URL: https://github.com/apache/beam/pull/11794#discussion_r432407631 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/package-info.java ## @@ -0,0 +1,2 @@ +/** Snowflake IO services and POJOs. */ Review comment: Thanks for spotting this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 438663) Time Spent: 1h 40m (was: 1.5h) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Assignee: Pawel Urbanowicz >Priority: P2 > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438300
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 28/May/20 15:10
Start Date: 28/May/20 15:10
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r431912755
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone expand(PC
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=438292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438292
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 28/May/20 15:01
Start Date: 28/May/20 15:01
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r430983300
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -172,10 +230,7 @@
abstract String getTable();
@Nullable
-abstract String getIntegrationName();
Review comment:
Not backwards compatible on a non-`@Experimental` class. Some of these
would be easy to implement on top of `Location` with a `@Deprecate` tag?
(My personal opinion would be not to worry too much about it... put
`@Experimental` on the SnowflakeIO?)
That being said, refactoring the two into a `Location` POJO doesn't really
add a LOT of value and it might be more worthwhile to put it back to two
separate configs. Up to you!
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/Location.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.Serializable;
+
+/**
+ * Used as one of the arguments for {@link
org.apache.beam.sdk.io.snowflake.SnowflakeIO} write and
+ * read operations. It keeps information about storage integration and staging
bucket name.
+ * Integration name is Snowflake storage integration object created according
to Snowflake
+ * documentation for the GCS bucket. Staging bucket name is Google Cloud
Storage bucket which in the
+ * case of writing operation will be used to save CSV files which will end up
in Snowflake under
+ * “staging_bucket_name/data” path and in the case of reading operation will
be used as a temporary
+ * location for storing CSV files named `sf_copy_csv_DATE_TIME_RANDOMSUFFIX`
which will be removed
+ * automatically once Read operation finishes.
+ */
+public class Location implements Serializable {
+ private String storageIntegrationName;
+ private String stagingBucketName;
+
+ public static Location of(SnowflakePipelineOptions options) {
+return new Location(options.getStorageIntegrationName(),
options.getStagingBucketName());
+ }
+
+ public static Location of(String storageIntegrationName, String
stagingBucketName) {
Review comment:
The original `withStagingBucketName` and `withIntegrationName` had some
useful javadoc and links!
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.i
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=437704&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437704
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 27/May/20 08:56
Start Date: 27/May/20 08:56
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r430886885
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.io.FileIO.Sink} for writing
CSV. */
+public class CSVSink implements FileIO.Sink {
Review comment:
Yes, we are using [COPY INTO
table](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#type-csv)
with CSV format.
Currently, `SnowflakeIO.write `is constructed in a way that requires a
particular table to exist in Snowflake before starting writing into Snowflake.
In one of the next Snowflake PR's we're planning to add the option for a
user a possibility for passing table schema.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437704)
Time Spent: 1h 10m (was: 1h)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Priority: P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=437676&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437676
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 27/May/20 06:34
Start Date: 27/May/20 06:34
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r430886885
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.io.FileIO.Sink} for writing
CSV. */
+public class CSVSink implements FileIO.Sink {
Review comment:
Yes, we are using [COPY INTO
table](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#type-csv)
with CSV format.
`SnowflakeIO.write `is constructed in a way that requires a particular table
to exist in Snowflake before starting writing into Snowflake.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437676)
Time Spent: 1h (was: 50m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Priority: P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=437228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437228
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 27/May/20 03:21
Start Date: 27/May/20 03:21
Worklog Time Spent: 10m
Work Description: pabloem commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r430612172
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/CSVSink.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.util.List;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+
+/** Implementation of {@link org.apache.beam.sdk.io.FileIO.Sink} for writing
CSV. */
+public class CSVSink implements FileIO.Sink {
Review comment:
I took only a very superficial look. Are you using CSV file imports to
write to Snowflake? I would be concerned about types being properly encoded in
the CSV (big integers, floats, bytes, date/time types, strings with
newlines/quotes). Have you already considered how to deal with all types?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437228)
Time Spent: 50m (was: 40m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=436476&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436476 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 22/May/20 10:39 Start Date: 22/May/20 10:39 Worklog Time Spent: 10m Work Description: purbanow commented on pull request #11794: URL: https://github.com/apache/beam/pull/11794#issuecomment-632624409 Hi @RyanSkraba will you find a moment to make a CR for this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 436476) Time Spent: 40m (was: 0.5h) > Add batch SnowflakeIO.Write to Java SDK > --- > > Key: BEAM-9894 > URL: https://issues.apache.org/jira/browse/BEAM-9894 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Dariusz Aniszewski >Priority: P2 > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=436448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436448
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 22/May/20 08:11
Start Date: 22/May/20 08:11
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r429105275
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeServiceImpl.java
##
@@ -1,90 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.snowflake;
-
-import java.sql.Connection;
-import java.sql.PreparedStatement;
-import java.sql.ResultSet;
-import java.sql.SQLException;
-import java.util.function.Consumer;
-import javax.sql.DataSource;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-
-/**
- * Implemenation of {@link org.apache.beam.sdk.io.snowflake.SnowflakeService}
used in production.
- */
-public class SnowflakeServiceImpl implements SnowflakeService {
Review comment:
Note: This file was moved to `services/SnowflakeServiceImpl.java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436448)
Time Spent: 0.5h (was: 20m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=436447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436447
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 22/May/20 08:11
Start Date: 22/May/20 08:11
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r429105275
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeServiceImpl.java
##
@@ -1,90 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.snowflake;
-
-import java.sql.Connection;
-import java.sql.PreparedStatement;
-import java.sql.ResultSet;
-import java.sql.SQLException;
-import java.util.function.Consumer;
-import javax.sql.DataSource;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-
-/**
- * Implemenation of {@link org.apache.beam.sdk.io.snowflake.SnowflakeService}
used in production.
- */
-public class SnowflakeServiceImpl implements SnowflakeService {
Review comment:
Note: This file moved to `services/SnowflakeServiceImpl.java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436447)
Time Spent: 20m (was: 10m)
> Add batch SnowflakeIO.Write to Java SDK
> ---
>
> Key: BEAM-9894
> URL: https://issues.apache.org/jira/browse/BEAM-9894
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Dariusz Aniszewski
>Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=436444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436444 ] ASF GitHub Bot logged work on BEAM-9894: Author: ASF GitHub Bot Created on: 22/May/20 08:05 Start Date: 22/May/20 08:05 Worklog Time Spent: 10m Work Description: purbanow opened a new pull request #11794: URL: https://github.com/apache/beam/pull/11794 This PR is part of adding SnowflakeIO to Java SDK [BEAM-9893](https://issues.apache.org/jira/browse/BEAM-9893). Precisely this PR is adding write operation to SnowflakeIO [BEAM-9894](https://issues.apache.org/jira/browse/BEAM-9894). The SnowflakeIO.Write works in the way that puts data on GCS as CSV files and then uses Snowflake's JDBC driver to run [COPY INTO TABLE](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html) statement to move CSV files from GCS to Snowflake table. As mentioned in the previous [PR](https://github.com/apache/beam/pull/11360), next PR’s will contain integration tests, streaming and cross-language support. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [