[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137590 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 21:39 Start Date: 23/Aug/18 21:39 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415579842 @iemejia - Thank you for all your feedback. I learned a few things in the process that I can now use within my group at GoDaddy. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137590) Time Spent: 5h (was: 4h 50m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: New Feature > Components: io-java-aws >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137589 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 21:39 Start Date: 23/Aug/18 21:39 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415579662 Forget to tell you, thanks a lot for contributing this one @jhalaria ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137589) Time Spent: 4h 50m (was: 4h 40m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: io-java-aws >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137588
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 23/Aug/18 21:37
Start Date: 23/Aug/18 21:37
Worklog Time Spent: 10m
Work Description: iemejia closed pull request #6202: [BEAM-4823] - Adds a
Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/io/amazon-web-services/build.gradle
b/sdks/java/io/amazon-web-services/build.gradle
index 3ea755966cd..23ed8b65e43 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -30,8 +30,11 @@ dependencies {
compile library.java.guava
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
shadow "com.amazonaws:aws-java-sdk-core:$aws_java_sdk_version"
+ shadow "com.amazonaws:aws-java-sdk-cloudwatch:$aws_java_sdk_version"
shadow "com.amazonaws:aws-java-sdk-s3:$aws_java_sdk_version"
shadow "com.amazonaws:aws-java-sdk-sqs:$aws_java_sdk_version"
+ shadow "com.amazonaws:aws-java-sdk-sns:$aws_java_sdk_version"
+ shadow "commons-lang:commons-lang:2.6"
shadow library.java.jackson_core
shadow library.java.jackson_annotations
shadow library.java.jackson_databind
@@ -39,6 +42,8 @@ dependencies {
runtime 'commons-codec:commons-codec:1.9'
runtime "org.apache.httpcomponents:httpclient:4.5.6"
testCompile project(path: ":beam-runners-direct-java", configuration:
"shadow")
+ testCompile project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ testCompile project(path: ":beam-sdks-java-io-common", configuration:
"shadow")
shadowTest library.java.guava_testlib
shadowTest library.java.hamcrest_core
shadowTest library.java.hamcrest_library
diff --git
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/AwsClientsProvider.java
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/AwsClientsProvider.java
new file mode 100644
index 000..dd11daa6431
--- /dev/null
+++
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/AwsClientsProvider.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.sns.AmazonSNS;
+import java.io.Serializable;
+
+/**
+ * Provides instances of AWS clients.
+ *
+ * Please note, that any instance of {@link AwsClientsProvider} must be
{@link Serializable} to
+ * ensure it can be sent to worker machines.
+ */
+public interface AwsClientsProvider extends Serializable {
+ AmazonCloudWatch getCloudWatchClient();
+
+ AmazonSNS createSnsPublisher();
+}
diff --git
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
new file mode 100644
index 000..c2481fc8bcc
--- /dev/null
+++
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137485 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 17:36 Start Date: 23/Aug/18 17:36 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415504796 @iemejia - All checks now seem to pass. Did you do some magic? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137485) Time Spent: 4.5h (was: 4h 20m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137441 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 16:09 Start Date: 23/Aug/18 16:09 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415474278 @jhalaria I am still puzzled let's see if this is a CI issue, I tried this locally and everything passes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137441) Time Spent: 4h 20m (was: 4h 10m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137440 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 16:08 Start Date: 23/Aug/18 16:08 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415473961 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137440) Time Spent: 4h 10m (was: 4h) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137231 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 23/Aug/18 01:10 Start Date: 23/Aug/18 01:10 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415239022 @iemejia - I rebased. Build is still failing & I don't see errors. Do you have any recommendations on how to figure this out? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137231) Time Spent: 4h (was: 3h 50m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137141 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 22/Aug/18 20:21 Start Date: 22/Aug/18 20:21 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415166566 Oups I somehow screwed this one while merging another PR can you please rebase/fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 137141) Time Spent: 3h 50m (was: 3h 40m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136988 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 22/Aug/18 15:16 Start Date: 22/Aug/18 15:16 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-415069251 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136988) Time Spent: 3h 40m (was: 3.5h) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136762 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 21/Aug/18 22:27 Start Date: 21/Aug/18 22:27 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-414841993 @iemejia - I have always seen ```Aug 21, 2018 2:19:08 PM org.apache.beam.sdk.io.aws.s3.S3FileSystem INFO: The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message. ``` in my local logs. But in the build server this is flagged as an error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136762) Time Spent: 3.5h (was: 3h 20m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136747 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 21/Aug/18 22:05 Start Date: 21/Aug/18 22:05 Worklog Time Spent: 10m Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-414837313 Mmm weird, tests aren't happy now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136747) Time Spent: 3h 20m (was: 3h 10m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136731
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 21:21
Start Date: 21/Aug/18 21:21
Worklog Time Spent: 10m
Work Description: jhalaria commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211764289
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
Review comment:
Fixed the javadoc validation error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136731)
Time Spent: 3h (was: 2h 50m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136732 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 21/Aug/18 21:21 Start Date: 21/Aug/18 21:21 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-414825873 @iemejia - I updated the PR incorporating your feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136732) Time Spent: 3h 10m (was: 3h) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136568
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211640247
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
+ * name of the SNS topic you're going to write to
+ * the max number of retries to perform.
+ * maximum total delay
+ * delay between retries
+ * need to specify AwsClientsProvider. You can pass on the default one
BasicSnsProvider
+ * an output tag where you can get results. Example in SnsIOTest
+ *
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ private static final int DEFAULT_MAX_RETRIES = 6;
+
+ //Write data tp SNS
+ public static Write write() {
+return new
AutoValue_SnsIO_Write.Builder().setMaxRetries(DEFAULT_MAX_RETRIES).build();
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write
+ extends PTransform, PCollectionTuple> {
+@Nullable
+abstract String getTopicName();
+
+@Nullable
+abstract AwsClientsProvider getAWSClientsProvider();
+
+@Nullable
+abstract Duration getMaxDelay();
+
+@Nonnull
+abstract int getMaxRetries();
+
+@Nullable
+abstract Duration getRetryDelay();
+
+@Nullable
+abstract TupleTag getResultOutputTag();
+
+abstract Builder builder();
+
+@AutoValue.Builder
+abstract static class Builder {
+
+ abstract Builder setTopicName(String topicName);
+
+ abstract Builder setAWSClientsProvider(AwsClientsProvider
clientProvider);
+
+ abstract Builder setMaxDelay(Duration maxDelay);
+
+ abstract Builder setMaxRetries(int maxRetries);
+
+ abstract Builder setRetryDelay(Duration retryDelay);
+
+ abstract Builder setResultOutputTag(TupleTag results);
+
+ abstract
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136570
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211646489
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
+ * name of the SNS topic you're going to write to
+ * the max number of retries to perform.
+ * maximum total delay
+ * delay between retries
+ * need to specify AwsClientsProvider. You can pass on the default one
BasicSnsProvider
+ * an output tag where you can get results. Example in SnsIOTest
+ *
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ private static final int DEFAULT_MAX_RETRIES = 6;
+
+ //Write data tp SNS
+ public static Write write() {
+return new
AutoValue_SnsIO_Write.Builder().setMaxRetries(DEFAULT_MAX_RETRIES).build();
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write
+ extends PTransform, PCollectionTuple> {
+@Nullable
+abstract String getTopicName();
+
+@Nullable
+abstract AwsClientsProvider getAWSClientsProvider();
+
+@Nullable
+abstract Duration getMaxDelay();
+
+@Nonnull
+abstract int getMaxRetries();
+
+@Nullable
+abstract Duration getRetryDelay();
+
+@Nullable
+abstract TupleTag getResultOutputTag();
+
+abstract Builder builder();
+
+@AutoValue.Builder
+abstract static class Builder {
+
+ abstract Builder setTopicName(String topicName);
+
+ abstract Builder setAWSClientsProvider(AwsClientsProvider
clientProvider);
+
+ abstract Builder setMaxDelay(Duration maxDelay);
+
+ abstract Builder setMaxRetries(int maxRetries);
+
+ abstract Builder setRetryDelay(Duration retryDelay);
+
+ abstract Builder setResultOutputTag(TupleTag results);
+
+ abstract
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136566
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211639088
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
Review comment:
Javadoc validation seems not to be happy about this one, please ensure the
tests pass locally via `./gradlew -p sdks/java/io/amazon-web-services clean
build`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136566)
Time Spent: 2h 20m (was: 2h 10m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136565
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211639736
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.AmazonSNSClientBuilder;
+import javax.annotation.Nullable;
+
+/** Basic implementation of {@link AwsClientsProvider} used by default in
{@link SnsIO}. */
+public class BasicSnsProvider implements AwsClientsProvider {
Review comment:
make package private
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136565)
Time Spent: 2h 10m (was: 2h)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136569
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211648313
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
+ * name of the SNS topic you're going to write to
+ * the max number of retries to perform.
+ * maximum total delay
+ * delay between retries
+ * need to specify AwsClientsProvider. You can pass on the default one
BasicSnsProvider
+ * an output tag where you can get results. Example in SnsIOTest
+ *
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ private static final int DEFAULT_MAX_RETRIES = 6;
+
+ //Write data tp SNS
+ public static Write write() {
+return new
AutoValue_SnsIO_Write.Builder().setMaxRetries(DEFAULT_MAX_RETRIES).build();
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write
+ extends PTransform, PCollectionTuple> {
+@Nullable
+abstract String getTopicName();
+
+@Nullable
+abstract AwsClientsProvider getAWSClientsProvider();
+
+@Nullable
+abstract Duration getMaxDelay();
+
+@Nonnull
+abstract int getMaxRetries();
+
+@Nullable
+abstract Duration getRetryDelay();
+
+@Nullable
+abstract TupleTag getResultOutputTag();
+
+abstract Builder builder();
+
+@AutoValue.Builder
+abstract static class Builder {
+
+ abstract Builder setTopicName(String topicName);
+
+ abstract Builder setAWSClientsProvider(AwsClientsProvider
clientProvider);
+
+ abstract Builder setMaxDelay(Duration maxDelay);
+
+ abstract Builder setMaxRetries(int maxRetries);
+
+ abstract Builder setRetryDelay(Duration retryDelay);
+
+ abstract Builder setResultOutputTag(TupleTag results);
+
+ abstract
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136567
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211640630
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.model.GetTopicAttributesResult;
+import com.amazonaws.services.sns.model.PublishRequest;
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import net.jodah.failsafe.Failsafe;
+import net.jodah.failsafe.RetryPolicy;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s for writing to https://aws.amazon.com/sns/;>SNS.
+ *
+ * Writing to SNS
+ *
+ * Example usage:
+ *
+ * {@code
+ * PCollection data = ...;
+ *
+ * data.apply(SnsIO.write()
+ * .withTopicName("topicName")
+ * .withMaxRetries(MAX_RETRIES) //eg. 2
+ * .withMaxDelay(MAX_DELAY) //eg. Duration.ofSeconds(2)
+ * .withRetryDelay(RETRY_DELAY) //eg. Duration.ofSeconds(1)
+ * .withAWSClientsProvider(new Provider(new AmazonSNSMock()))
+ * .withResultOutputTag(results));
+ * }
+ *
+ * As a client, you need to provide at least the following things:
+ *
+ *
+ * AwsClientsProvider
+ * name of the SNS topic you're going to write to
+ * the max number of retries to perform.
+ * maximum total delay
+ * delay between retries
+ * need to specify AwsClientsProvider. You can pass on the default one
BasicSnsProvider
+ * an output tag where you can get results. Example in SnsIOTest
+ *
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ private static final int DEFAULT_MAX_RETRIES = 6;
+
+ //Write data tp SNS
+ public static Write write() {
+return new
AutoValue_SnsIO_Write.Builder().setMaxRetries(DEFAULT_MAX_RETRIES).build();
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write
+ extends PTransform, PCollectionTuple> {
+@Nullable
+abstract String getTopicName();
+
+@Nullable
+abstract AwsClientsProvider getAWSClientsProvider();
+
+@Nullable
+abstract Duration getMaxDelay();
+
+@Nonnull
+abstract int getMaxRetries();
+
+@Nullable
+abstract Duration getRetryDelay();
+
+@Nullable
+abstract TupleTag getResultOutputTag();
+
+abstract Builder builder();
+
+@AutoValue.Builder
+abstract static class Builder {
+
+ abstract Builder setTopicName(String topicName);
+
+ abstract Builder setAWSClientsProvider(AwsClientsProvider
clientProvider);
+
+ abstract Builder setMaxDelay(Duration maxDelay);
+
+ abstract Builder setMaxRetries(int maxRetries);
+
+ abstract Builder setRetryDelay(Duration retryDelay);
+
+ abstract Builder setResultOutputTag(TupleTag results);
+
+ abstract
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136571
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 21/Aug/18 15:32
Start Date: 21/Aug/18 15:32
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211639508
##
File path: sdks/java/io/amazon-web-services/build.gradle
##
@@ -29,6 +29,10 @@ dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
shadow "com.amazonaws:aws-java-sdk-core:$aws_java_sdk_version"
shadow "com.amazonaws:aws-java-sdk-s3:$aws_java_sdk_version"
+ shadow "com.amazonaws:aws-java-sdk-sns:$aws_java_sdk_version"
+ shadow "com.amazonaws:aws-java-sdk-cloudwatch:$aws_java_sdk_version"
+ shadow "commons-lang:commons-lang:2.6"
+ shadow group: 'net.jodah', name: 'failsafe', version: '1.1.0'
Review comment:
Remove this dependency by replacing it with internal Beam Backoff impl, see
more details below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136571)
Time Spent: 2h 50m (was: 2h 40m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136250 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 20/Aug/18 20:04 Start Date: 20/Aug/18 20:04 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-41948 @iemejia - Thank you for looking at the PR. I have addressed your feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136250) Time Spent: 2h (was: 1h 50m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136247
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 20/Aug/18 19:56
Start Date: 20/Aug/18 19:56
Worklog Time Spent: 10m
Work Description: jhalaria commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211387705
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/PublishResultCoder.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.sns;
+
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.common.collect.ImmutableList;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+
+/** Custom Coder for handling publish result. */
+public class PublishResultCoder extends Coder {
Review comment:
`PublushResult` is `Serializable` but it extends a class `ResponseMetadata`
which isn't. I have added a `SnsCoderProviderRegistrar` to auto register the
class. Thanks for providing the reference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136247)
Time Spent: 1.5h (was: 1h 20m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136249 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 20/Aug/18 19:56 Start Date: 20/Aug/18 19:56 Worklog Time Spent: 10m Work Description: jhalaria commented on a change in pull request #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#discussion_r211387779 ## File path: sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/AwsClientsProvider.java ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.io.sns; Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136249) Time Spent: 1h 50m (was: 1h 40m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136248
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 20/Aug/18 19:56
Start Date: 20/Aug/18 19:56
Worklog Time Spent: 10m
Work Description: jhalaria commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211387758
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/DefaultSnsClientProvider.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.sns;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.AmazonSNSClientBuilder;
+
+/**
+ * Provides default provider implementation for SnS.
+ *
+ * Please note, that any instance of {@link AwsClientsProvider} must be
{@link
+ * java.io.Serializable} to ensure it can be sent to worker machines.
+ */
+public class DefaultSnsClientProvider implements AwsClientsProvider {
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136248)
Time Spent: 1h 40m (was: 1.5h)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136158
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 20/Aug/18 16:05
Start Date: 20/Aug/18 16:05
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211318561
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/DefaultSnsClientProvider.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.sns;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.AmazonSNSClientBuilder;
+
+/**
+ * Provides default provider implementation for SnS.
+ *
+ * Please note, that any instance of {@link AwsClientsProvider} must be
{@link
+ * java.io.Serializable} to ensure it can be sent to worker machines.
+ */
+public class DefaultSnsClientProvider implements AwsClientsProvider {
Review comment:
This implies also adding the configuration via withXXX methods in the write
to support authentication (sorry if I was not clear about that).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136158)
Time Spent: 1h 20m (was: 1h 10m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136155
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 20/Aug/18 16:00
Start Date: 20/Aug/18 16:00
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r211314296
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/PublishResultCoder.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.sns;
+
+import com.amazonaws.services.sns.model.PublishResult;
+import com.google.common.collect.ImmutableList;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+
+/** Custom Coder for handling publish result. */
+public class PublishResultCoder extends Coder {
Review comment:
I was wondering if given that PublishResult is supposedly `Serializable` if
we could get rid of this one altogether. Can we?
If this is not possible, can you please create a `SnsCoderProviderRegistrar`
to automatically register the Class with the valid Coder. See
[this](https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java#L30)
for ref. This way users don't have to do it manually (or here in the test)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136155)
Time Spent: 1h (was: 50m)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136156
]
ASF GitHub Bot logged work on BEAM-4823:
Author: ASF GitHub Bot
Created on: 20/Aug/18 16:00
Start Date: 20/Aug/18 16:00
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6202:
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r209998398
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/DefaultSnsClientProvider.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.sns;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
+import com.amazonaws.services.sns.AmazonSNS;
+import com.amazonaws.services.sns.AmazonSNSClientBuilder;
+
+/**
+ * Provides default provider implementation for SnS.
+ *
+ * Please note, that any instance of {@link AwsClientsProvider} must be
{@link
+ * java.io.Serializable} to ensure it can be sent to worker machines.
+ */
+public class DefaultSnsClientProvider implements AwsClientsProvider {
Review comment:
Can you please rename this class to `BasicSnsProvider` and add the
authentication configuration parameters akin to `BasicKinesisProvider`
(accessKey, secretKey, region, serviceEndpoint).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 136156)
Time Spent: 1h 10m (was: 1h)
> Create SNS IO
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=136154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136154 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 20/Aug/18 16:00 Start Date: 20/Aug/18 16:00 Worklog Time Spent: 10m Work Description: iemejia commented on a change in pull request #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#discussion_r209777201 ## File path: sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/sns/AwsClientsProvider.java ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.io.sns; Review comment: Please move the files / package to be inside of aws: `org.apache.beam.sdk.io.aws.sns` instead of `org.apache.beam.sdk.io.sns` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 136154) Time Spent: 50m (was: 40m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=134063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-134063 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 13/Aug/18 06:52 Start Date: 13/Aug/18 06:52 Worklog Time Spent: 10m Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#issuecomment-412424303 @iemejia - Thank you for looking at the PR. I moved `SNS` changes to `sdks/java/io/amazon-web-services` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 134063) Time Spent: 40m (was: 0.5h) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=134062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-134062 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 13/Aug/18 06:51 Start Date: 13/Aug/18 06:51 Worklog Time Spent: 10m Work Description: jhalaria commented on a change in pull request #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#discussion_r209505503 ## File path: sdks/java/io/sns/src/main/java/org/apache/beam/sdk/io/sns/AwsClientsProvider.java ## @@ -0,0 +1,17 @@ +package org.apache.beam.sdk.io.sns; Review comment: Added Apache headers to all files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 134062) Time Spent: 0.5h (was: 20m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=133825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133825 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 10/Aug/18 21:39 Start Date: 10/Aug/18 21:39 Worklog Time Spent: 10m Work Description: iemejia commented on a change in pull request #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202#discussion_r209390115 ## File path: sdks/java/io/sns/src/main/java/org/apache/beam/sdk/io/sns/AwsClientsProvider.java ## @@ -0,0 +1,17 @@ +package org.apache.beam.sdk.io.sns; Review comment: Missing Apache headers in this and other files (that's the reason the build broke) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 133825) Time Spent: 20m (was: 10m) > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Jean-Baptiste Onofré >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4823) Create SNS IO
[ https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=133705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133705 ] ASF GitHub Bot logged work on BEAM-4823: Author: ASF GitHub Bot Created on: 10/Aug/18 17:34 Start Date: 10/Aug/18 17:34 Worklog Time Spent: 10m Work Description: jhalaria opened a new pull request #6202: [BEAM-4823] - Adds a Sink to write to Amazon's SNS URL: https://github.com/apache/beam/pull/6202 Adds a new Sink to write Amazon's SNS [a.k.a. Simple Notification Service]. It uses the AmazonSNS client underneath to submit requests. Adds a retry logic to re-publish requests when writes to SNS failed due to service not available. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 133705) Time Spent: 10m Remaining Estimate: 0h > Create SNS IO > -- > > Key: BEAM-4823 > URL: https://issues.apache.org/jira/browse/BEAM-4823 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: Not applicable >Reporter: Ankit Jhalaria >Assignee: Jean-Baptiste Onofré >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
