[beam] branch asf-site updated: Publishing website 2019/06/14 20:34:41 at commit 4d2eba2
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 7dde544 Publishing website 2019/06/14 20:34:41 at commit 4d2eba2 7dde544 is described below commit 7dde544d367cefff8d27d0ce13212855661309c7 Author: jenkins AuthorDate: Fri Jun 14 20:34:41 2019 + Publishing website 2019/06/14 20:34:41 at commit 4d2eba2 --- website/generated-content/contribute/design-documents/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index ee2aaef..7d563b3 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -407,6 +407,7 @@ limitations under the License. Splittable DoFn for Python SDK [http://s.apache.org/splittable-do-fn-python-sdk;>doc] Parquet IO for Python SDK [https://docs.google.com/document/d/1-FT6zmjYhYFWXL8aDM5mNeiUnZdKnnB021zTo4S-0Wg;>doc] Building Python Wheels [https://docs.google.com/document/d/1MRVFs48e6g7wORshr2UpuOVD_yTSJTbmR65_j8XbGek;>doc] + Beam Type Hints for Python 3 [https://docs.google.com/document/d/15bsOL3YcUWuIjnxqhi9nanhj2eh9S6-QlLYuL7ufcXY;>doc] Go
[beam] branch master updated (19804ac -> 4d2eba2)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 19804ac Merge pull request #8718: [BEAM-7450] Support unbounded reads with HCatalogIO add 2d49e58 Add link to type hints design doc new 4d2eba2 Merge pull request #8865: [website] Add link to the type hints design doc for python 3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: Merge pull request #8865: [website] Add link to the type hints design doc for python 3
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4d2eba2dd82a9f27bc012195ad321dc4d51a7096 Merge: 19804ac 2d49e58 Author: Ismaël Mejía AuthorDate: Fri Jun 14 22:33:39 2019 +0200 Merge pull request #8865: [website] Add link to the type hints design doc for python 3 website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated: [BEAM-7450] Support unbounded reads with HCatalogIO
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9bbe6f5 [BEAM-7450] Support unbounded reads with HCatalogIO
new 19804ac Merge pull request #8718: [BEAM-7450] Support unbounded reads
with HCatalogIO
9bbe6f5 is described below
commit 9bbe6f523aac427958a4e99e4d729a80b105e63d
Author: Ankit Jhalaria
AuthorDate: Wed May 29 12:33:01 2019 -0700
[BEAM-7450] Support unbounded reads with HCatalogIO
---
.../apache/beam/sdk/io/hcatalog/HCatalogIO.java| 91 +++--
.../apache/beam/sdk/io/hcatalog/HCatalogUtils.java | 87
.../beam/sdk/io/hcatalog/PartitionPollerFn.java| 56 +++
.../beam/sdk/io/hcatalog/PartitionReaderFn.java| 111 +
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 80 +++
5 files changed, 414 insertions(+), 11 deletions(-)
diff --git
a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
index 73518f6..05b43c6 100644
---
a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
+++
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
@@ -25,7 +25,6 @@ import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
import java.util.NoSuchElementException;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
@@ -33,15 +32,17 @@ import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.hadoop.WritableCoder;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Watch;
+import org.apache.beam.sdk.transforms.Watch.Growth.TerminationCondition;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import
org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.ql.metadata.Table;
@@ -58,6 +59,7 @@ import org.apache.hive.hcatalog.data.transfer.ReadEntity;
import org.apache.hive.hcatalog.data.transfer.ReaderContext;
import org.apache.hive.hcatalog.data.transfer.WriteEntity;
import org.apache.hive.hcatalog.data.transfer.WriterContext;
+import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -83,6 +85,20 @@ import org.slf4j.LoggerFactory;
* .withFilter(filterString) //optional, may be specified if the table
is partitioned
* }
*
+ * HCatalog source supports reading of HCatRecord in an unbounded mode.
When run in an unbounded
+ * mode, HCatalogIO will continuously poll for new partitions and read that
data. If provided with a
+ * termination condition, it will stop reading data after the condition is met.
+ *
+ * {@code
+ * pipeline
+ * .apply(HCatalogIO.read()
+ * .withConfigProperties(configProperties)
+ * .withDatabase("default") //optional, assumes default if none specified
+ * .withTable("employee")
+ * .withPollingInterval(Duration.millis(15000)) // poll for new
partitions every 15 seconds
+ *
.withTerminationCondition(Watch.Growth.afterTotalOf(Duration.millis(6
//optional
+ * }
+ *
* Writing using HCatalog
*
* HCatalog sink supports writing of HCatRecord to a HCatalog managed
source, for eg. Hive.
@@ -120,7 +136,10 @@ public class HCatalogIO {
/** Read data from Hive. */
public static Read read() {
-return new
AutoValue_HCatalogIO_Read.Builder().setDatabase(DEFAULT_DATABASE).build();
+return new AutoValue_HCatalogIO_Read.Builder()
+.setDatabase(DEFAULT_DATABASE)
+.setPartitionCols(new ArrayList<>())
+.build();
}
private HCatalogIO() {}
@@ -129,6 +148,7 @@ public class HCatalogIO {
@VisibleForTesting
@AutoValue
public abstract static class Read extends PTransform> {
+
@Nullable
abstract Map getConfigProperties();
@@ -147,6 +167,15 @@ public class HCatalogIO {
@Nullable
abstract Integer getSplitId();
+@Nullable
+abstract Duration getPollingInterval();
+
+@Nullable
+abstract List getPartitionCols();
+
+@Nullable
+abstract TerminationCondition getTerminationCondition();
+
abstract Builder
[beam] branch udim-patch-1 created (now 2d49e58)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch udim-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at 2d49e58 Add link to type hints design doc This branch includes the following new commits: new 2d49e58 Add link to type hints design doc The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Add link to type hints design doc
This is an automated email from the ASF dual-hosted git repository. udim pushed a commit to branch udim-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git commit 2d49e586ac716bd356e707b02c318bfd2dfef5c2 Author: Udi Meiri AuthorDate: Fri Jun 14 13:16:54 2019 -0700 Add link to type hints design doc --- website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/src/contribute/design-documents.md b/website/src/contribute/design-documents.md index 36fdb31..6c3e159 100644 --- a/website/src/contribute/design-documents.md +++ b/website/src/contribute/design-documents.md @@ -149,6 +149,7 @@ This is a collection of documents that may or may not be up to date. - Splittable DoFn for Python SDK [[doc](http://s.apache.org/splittable-do-fn-python-sdk)] - Parquet IO for Python SDK [[doc](https://docs.google.com/document/d/1-FT6zmjYhYFWXL8aDM5mNeiUnZdKnnB021zTo4S-0Wg)] - Building Python Wheels [[doc](https://docs.google.com/document/d/1MRVFs48e6g7wORshr2UpuOVD_yTSJTbmR65_j8XbGek)] +- Beam Type Hints for Python 3 [[doc](https://docs.google.com/document/d/15bsOL3YcUWuIjnxqhi9nanhj2eh9S6-QlLYuL7ufcXY)] ### Go - Apache Beam Go SDK design [[doc](https://s.apache.org/beam-go-sdk-design-rfc)]
[beam] 01/01: Merge pull request #8765: [BEAM-7432] Set input and output files for sdist
This is an automated email from the ASF dual-hosted git repository. udim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a71bfda77df36aa1531f01533c372233cfba0dd9 Merge: 6b3b970 b25be12 Author: Udi Meiri AuthorDate: Fri Jun 14 21:18:34 2019 +0300 Merge pull request #8765: [BEAM-7432] Set input and output files for sdist .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (6b3b970 -> a71bfda)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6b3b970 Merge pull request #8831: Reduce gradle build verbosity add b25be12 [BEAM-7432] Set input and output files for sdist new a71bfda Merge pull request #8765: [BEAM-7432] Set input and output files for sdist The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (11aacd2 -> 6b3b970)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 11aacd2 Remove unused **kwargs arguments for various transforms. add 9b99768 Reduce gradle build verbosity new 6b3b970 Merge pull request #8831: Reduce gradle build verbosity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 - 1 file changed, 1 deletion(-)
[beam] 01/01: Merge pull request #8831: Reduce gradle build verbosity
This is an automated email from the ASF dual-hosted git repository. udim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 6b3b9705768b777dc776dd25f35d99070bde95a4 Merge: 11aacd2 9b99768 Author: Udi Meiri AuthorDate: Fri Jun 14 21:03:59 2019 +0300 Merge pull request #8831: Reduce gradle build verbosity buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (e46d0f8 -> 11aacd2)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e46d0f8 [BEAM-7531] Remove and disallow KMS key setting add 55e3f16 Remove unused **kwargs arguments for various transforms. new 11aacd2 Remove unused **kwargs arguments for various transforms. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/transforms/util.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] 01/01: Remove unused **kwargs arguments for various transforms.
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 11aacd2ea2e2b1fec54efc0557e016e2b7f8d944 Merge: e46d0f8 55e3f16 Author: Lukasz Cwik AuthorDate: Fri Jun 14 10:27:07 2019 -0700 Remove unused **kwargs arguments for various transforms. sdks/python/apache_beam/transforms/util.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (2be7457 -> e46d0f8)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2be7457 Adding a Link to The Doc for Cost Estimation add 5b3807b [BEAM-7531] Remove and disallow KMS key setting new e46d0f8 [BEAM-7531] Remove and disallow KMS key setting The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/extensions/gcp/options/GcpOptions.java | 11 +-- .../beam/sdk/extensions/gcp/options/GcpOptionsTest.java| 14 -- 2 files changed, 17 insertions(+), 8 deletions(-)
[beam] 01/01: [BEAM-7531] Remove and disallow KMS key setting
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e46d0f8d17a107190aecd391885402f53f0e10c2 Merge: 2be7457 5b3807b Author: Lukasz Cwik AuthorDate: Fri Jun 14 10:07:52 2019 -0700 [BEAM-7531] Remove and disallow KMS key setting .../apache/beam/sdk/extensions/gcp/options/GcpOptions.java | 11 +-- .../beam/sdk/extensions/gcp/options/GcpOptionsTest.java| 14 -- 2 files changed, 17 insertions(+), 8 deletions(-)
[beam] branch asf-site updated: Publishing website 2019/06/14 16:52:11 at commit 2be7457
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0fd8355 Publishing website 2019/06/14 16:52:11 at commit 2be7457 0fd8355 is described below commit 0fd835581581ac6825d60849875d779362fdbafe Author: jenkins AuthorDate: Fri Jun 14 16:52:12 2019 + Publishing website 2019/06/14 16:52:11 at commit 2be7457 --- website/generated-content/contribute/design-documents/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index c36e4f0..ee2aaef 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -345,6 +345,7 @@ limitations under the License. Beam SQL Pipeline Options [https://docs.google.com/document/d/1UTsSBuruJRfGnVOS9eXbQI6NauCD4WnSAPgA_Y0zjdk;>doc] Unbounded limit [https://docs.google.com/document/d/13zeTewHH9nfwhSlcE4x77WQwr1U2Z4sTiNRjOXUj2aw;>doc] Portable Beam Schemas [https://s.apache.org/beam-schemas;>doc] + Cost Estimation [https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY/edit#heading=h.6rlkpwwx7gvf;>doc] Portability
[beam] 01/01: Adding a Link to The Doc for Cost Estimation
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2be7457a4c0b311c3bd784b3f00b425596adeb06 Merge: d38a4da f5bc2fe Author: Lukasz Cwik AuthorDate: Fri Jun 14 09:49:14 2019 -0700 Adding a Link to The Doc for Cost Estimation website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (d38a4da -> 2be7457)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d38a4da Merge pull request #8858: [BEAM-7542] Fix faulty cast in BigQueryUtils add f5bc2fe Adding a Link to The Doc for Cost Estimation new 2be7457 Adding a Link to The Doc for Cost Estimation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/contribute/design-documents.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated: Merge pull request #8858: [BEAM-7542] Fix faulty cast in BigQueryUtils
This is an automated email from the ASF dual-hosted git repository.
gleb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d38a4da Merge pull request #8858: [BEAM-7542] Fix faulty cast in
BigQueryUtils
d38a4da is described below
commit d38a4da49d258509e15d8335547729cebfbbc5aa
Author: Viktor Gerdin
AuthorDate: Fri Jun 14 16:07:32 2019 +0200
Merge pull request #8858: [BEAM-7542] Fix faulty cast in BigQueryUtils
fix BYTES type conversion
---
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java| 7 +---
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 46 +++---
2 files changed, 34 insertions(+), 19 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
index 3ef1507..07102b3 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
@@ -27,7 +27,6 @@ import com.google.api.services.bigquery.model.TableSchema;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import java.math.BigDecimal;
-import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -123,6 +122,7 @@ public class BigQueryUtils {
str ->
new DateTime(
(long) (Double.parseDouble(str) * 1000),
ISOChronology.getInstanceUTC()))
+ .put(TypeName.BYTES, str -> BaseEncoding.base64().decode(str))
.build();
// TODO: BigQuery code should not be relying on Calcite metadata fields. If
so, this belongs
@@ -371,10 +371,7 @@ public class BigQueryUtils {
return fieldValue.toString();
case BYTES:
-ByteBuffer byteBuffer = (ByteBuffer) fieldValue;
-byte[] bytes = new byte[byteBuffer.limit()];
-byteBuffer.get(bytes);
-return BaseEncoding.base64().encode(bytes);
+return BaseEncoding.base64().encode((byte[]) fieldValue);
default:
return fieldValue;
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
index 2226be9..86b2d27 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
@@ -32,6 +32,7 @@ import
com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import java.util.Arrays;
+import java.util.Base64;
import java.util.Collections;
import java.util.List;
import
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.ConversionOptions.TruncateTimestamps;
@@ -54,6 +55,7 @@ public class BigQueryUtilsTest {
.addNullableField("name", Schema.FieldType.STRING)
.addNullableField("timestamp", Schema.FieldType.DATETIME)
.addNullableField("valid", Schema.FieldType.BOOLEAN)
+ .addNullableField("binary", Schema.FieldType.BYTES)
.build();
private static final Schema ARRAY_TYPE =
@@ -80,6 +82,9 @@ public class BigQueryUtilsTest {
private static final TableFieldSchema VALID =
new
TableFieldSchema().setName("valid").setType(StandardSQLTypeName.BOOL.toString());
+ private static final TableFieldSchema BINARY =
+ new
TableFieldSchema().setName("binary").setType(StandardSQLTypeName.BYTES.toString());
+
private static final TableFieldSchema IDS =
new TableFieldSchema()
.setName("ids")
@@ -91,18 +96,25 @@ public class BigQueryUtilsTest {
.setName("row")
.setType(StandardSQLTypeName.STRUCT.toString())
.setMode(Mode.NULLABLE.toString())
- .setFields(Arrays.asList(ID, VALUE, NAME, TIMESTAMP, VALID));
+ .setFields(Arrays.asList(ID, VALUE, NAME, TIMESTAMP, VALID, BINARY));
private static final TableFieldSchema ROWS =
new TableFieldSchema()
.setName("rows")
.setType(StandardSQLTypeName.STRUCT.toString())
.setMode(Mode.REPEATED.toString())
- .setFields(Arrays.asList(ID, VALUE, NAME, TIMESTAMP, VALID));
+ .setFields(Arrays.asList(ID, VALUE, NAME, TIMESTAMP, VALID, BINARY));
+ // Make sure that chosen BYTES test value is the same after a full base64
round trip.
private static final Row FLAT_ROW =
Row.withSchema(FLAT_TYPE)
- .addValues(123L, 123.456, "test", new
[beam] branch master updated (59ea050 -> f2067fe)
This is an automated email from the ASF dual-hosted git repository. mxm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 59ea050 Merge pull request #8675: [BEAM-7402] BigQuery IO read performance tests add 64abb60 [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction new f2067fe Merge pull request #8857: [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../functions/ImpulseSourceFunction.java | 33 - .../functions/ImpulseSourceFunctionTest.java | 83 -- 2 files changed, 109 insertions(+), 7 deletions(-)
[beam] 01/01: Merge pull request #8857: [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction
This is an automated email from the ASF dual-hosted git repository. mxm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f2067fe89f75e269d2171afd6628480ce96ec617 Merge: 59ea050 64abb60 Author: Maximilian Michels AuthorDate: Fri Jun 14 15:04:18 2019 +0200 Merge pull request #8857: [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction .../functions/ImpulseSourceFunction.java | 33 - .../functions/ImpulseSourceFunctionTest.java | 83 -- 2 files changed, 109 insertions(+), 7 deletions(-)
[beam] branch master updated (211c788 -> 59ea050)
This is an automated email from the ASF dual-hosted git repository.
lgajowy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 211c788 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler
type variables to allow for proper key parameterization
add c3a0fda [BEAM-7402] Added a performance test for BigQuery IO read
new 59ea050 Merge pull request #8675: [BEAM-7402] BigQuery IO read
performance tests
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...ite_perf_test.py => bigquery_read_perf_test.py} | 82 +++---
1 file changed, 56 insertions(+), 26 deletions(-)
copy sdks/python/apache_beam/io/gcp/{bigquery_write_perf_test.py =>
bigquery_read_perf_test.py} (56%)
[beam] 01/01: Merge pull request #8675: [BEAM-7402] BigQuery IO read performance tests
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 59ea050adbed21a708b0c7d272149c66e23d9632 Merge: 211c788 c3a0fda Author: Łukasz Gajowy AuthorDate: Fri Jun 14 13:17:08 2019 +0200 Merge pull request #8675: [BEAM-7402] BigQuery IO read performance tests .../apache_beam/io/gcp/bigquery_read_perf_test.py | 137 + 1 file changed, 137 insertions(+)
[beam] 01/01: Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 211c7886e3f7cc7b8a98a60535f3908328212da1 Merge: 6d093f5 f4cf92d Author: Thomas Weise AuthorDate: Fri Jun 14 10:59:37 2019 +0200 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization .../streaming/ExecutableStageDoFnOperator.java | 21 +++-- .../state/InMemoryBagUserStateFactory.java | 17 - .../fnexecution/state/StateRequestHandlers.java | 10 +- .../fnexecution/control/RemoteExecutionTest.java| 19 ++- 4 files changed, 34 insertions(+), 33 deletions(-)
[beam] branch master updated (6d093f5 -> 211c788)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6d093f5 Merge pull request #8825: [BEAM-7533] Fix CoderRegistry for Float add f4cf92d [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization new 211c788 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../streaming/ExecutableStageDoFnOperator.java | 21 +++-- .../state/InMemoryBagUserStateFactory.java | 17 - .../fnexecution/state/StateRequestHandlers.java | 10 +- .../fnexecution/control/RemoteExecutionTest.java| 19 ++- 4 files changed, 34 insertions(+), 33 deletions(-)
[beam] branch master updated (33d7cd0 -> 6d093f5)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 33d7cd0 Merge pull request #8390: [BEAM-7043] Add DynamoDBIO
add 121df06 [BEAM-7533] Fix CoderRegistry for Float
add 6d093f5 Merge pull request #8825: [BEAM-7533] Fix CoderRegistry for
Float
No new revisions were added by this update.
Summary of changes:
.../org/apache/beam/sdk/coders/CoderRegistry.java | 3 +-
.../apache/beam/sdk/coders/CoderRegistryTest.java | 4 ++
...tringUtf8CoderTest.java => FloatCoderTest.java} | 44 +++---
3 files changed, 28 insertions(+), 23 deletions(-)
copy
sdks/java/core/src/test/java/org/apache/beam/sdk/coders/{StringUtf8CoderTest.java
=> FloatCoderTest.java} (73%)
[beam] branch master updated: [BEAM-7043] Add DynamoDBIO
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3fdf691 [BEAM-7043] Add DynamoDBIO
new 33d7cd0 Merge pull request #8390: [BEAM-7043] Add DynamoDBIO
3fdf691 is described below
commit 3fdf691763201a48deab21a9279c27375b63559e
Author: Cam Mach
AuthorDate: Mon Apr 15 16:19:18 2019 -0700
[BEAM-7043] Add DynamoDBIO
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
sdks/java/io/amazon-web-services/build.gradle | 2 +
.../sdk/io/aws/dynamodb/AttributeValueCoder.java | 166 +++
.../AttributeValueCoderProviderRegistrar.java | 37 ++
.../sdk/io/aws/dynamodb/AwsClientsProvider.java| 34 ++
.../sdk/io/aws/dynamodb/BasicDynamoDBProvider.java | 75 +++
.../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 536 +
.../beam/sdk/io/aws/dynamodb/package-info.java | 19 +
.../io/aws/dynamodb/AttributeValueCoderTest.java | 211
.../io/aws/dynamodb/AwsClientsProviderMock.java| 46 ++
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 213
.../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 168 +++
12 files changed, 1508 insertions(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 57c1ccf..6bc456f 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -424,6 +424,7 @@ class BeamModulePlugin implements Plugin {
avro_tests :
"org.apache.avro:avro:1.8.2:tests",
aws_java_sdk_cloudwatch :
"com.amazonaws:aws-java-sdk-cloudwatch:$aws_java_sdk_version",
aws_java_sdk_core :
"com.amazonaws:aws-java-sdk-core:$aws_java_sdk_version",
+aws_java_sdk_dynamodb :
"com.amazonaws:aws-java-sdk-dynamodb:$aws_java_sdk_version",
aws_java_sdk_kinesis:
"com.amazonaws:aws-java-sdk-kinesis:$aws_java_sdk_version",
aws_java_sdk_s3 :
"com.amazonaws:aws-java-sdk-s3:$aws_java_sdk_version",
aws_java_sdk_sns:
"com.amazonaws:aws-java-sdk-sns:$aws_java_sdk_version",
@@ -565,7 +566,6 @@ class BeamModulePlugin implements Plugin {
url(project.properties['distMgmtSnapshotsUrl'] ?: isRelease(project)
?
'https://repository.apache.org/service/local/staging/deploy/maven2'
:
'https://repository.apache.org/content/repositories/snapshots')
-
// We attempt to find and load credentials from ~/.m2/settings.xml
file that a user
// has configured with the Apache release and snapshot staging
credentials.
//
diff --git a/sdks/java/io/amazon-web-services/build.gradle
b/sdks/java/io/amazon-web-services/build.gradle
index 560e4c5..0bf33dc 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -29,6 +29,7 @@ dependencies {
compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.aws_java_sdk_cloudwatch
compile library.java.aws_java_sdk_core
+ compile library.java.aws_java_sdk_dynamodb
compile library.java.aws_java_sdk_s3
compile library.java.aws_java_sdk_sns
compile library.java.aws_java_sdk_sqs
@@ -47,6 +48,7 @@ dependencies {
testCompile library.java.mockito_core
testCompile library.java.junit
testCompile 'org.elasticmq:elasticmq-rest-sqs_2.12:0.14.1'
+ testCompile 'org.testcontainers:localstack:1.11.2'
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(":runners:direct-java")
}
diff --git
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
new file mode 100644
index 000..4bdf8b5
--- /dev/null
+++
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the
[beam] 01/01: Merge pull request #8854: [BEAM-7553] Add portable runner to Spark webpage
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 84e54af84a4002eb88d755fc4843d3337f53ac1a Merge: 82bf1ae 614dad7 Author: Ismaël Mejía AuthorDate: Fri Jun 14 09:27:36 2019 +0200 Merge pull request #8854: [BEAM-7553] Add portable runner to Spark webpage website/src/documentation/runners/spark.md | 132 +++-- 1 file changed, 123 insertions(+), 9 deletions(-)
[beam] branch master updated (82bf1ae -> 84e54af)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 82bf1ae Merge pull request #8817: [BEAM-7530] allow read None bytes add 614dad7 [BEAM-7553] add portability to Spark webpage new 84e54af Merge pull request #8854: [BEAM-7553] Add portable runner to Spark webpage The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/documentation/runners/spark.md | 132 +++-- 1 file changed, 123 insertions(+), 9 deletions(-)
