[GitHub] incubator-beam pull request #739: [BEAM-484] Datastore Read as a composite P...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/739


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-484) Datastore Source should support Dynamic Splitting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403404#comment-15403404
 ] 

ASF GitHub Bot commented on BEAM-484:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/739


> Datastore Source should support Dynamic Splitting
> -
>
> Key: BEAM-484
> URL: https://issues.apache.org/jira/browse/BEAM-484
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Datastore Read as a composite PTransform

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8b1e64a66 -> bef1a69f1


Datastore Read as a composite PTransform


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/92030133
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/92030133
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/92030133

Branch: refs/heads/master
Commit: 92030133a721d0bcbd17f5751333b0449257332e
Parents: 8b1e64a
Author: Vikas Kedigehalli 
Authored: Tue Jul 26 09:54:43 2016 -0700
Committer: Dan Halperin 
Committed: Mon Aug 1 22:44:25 2016 -0700

--
 .../beam/sdk/io/gcp/datastore/V1Beta3.java  | 816 +--
 .../beam/sdk/io/gcp/datastore/V1Beta3Test.java  | 466 +--
 2 files changed, 615 insertions(+), 667 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/92030133/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3.java
index 0ba4433..bda907a 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3.java
@@ -24,7 +24,6 @@ import static com.google.common.base.Verify.verify;
 import static com.google.datastore.v1beta3.PropertyFilter.Operator.EQUAL;
 import static com.google.datastore.v1beta3.PropertyOrder.Direction.DESCENDING;
 import static 
com.google.datastore.v1beta3.QueryResultBatch.MoreResultsType.NOT_FINISHED;
-import static 
com.google.datastore.v1beta3.client.DatastoreHelper.makeAndFilter;
 import static com.google.datastore.v1beta3.client.DatastoreHelper.makeFilter;
 import static com.google.datastore.v1beta3.client.DatastoreHelper.makeOrder;
 import static com.google.datastore.v1beta3.client.DatastoreHelper.makeUpsert;
@@ -33,16 +32,22 @@ import static 
com.google.datastore.v1beta3.client.DatastoreHelper.makeValue;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.SerializableCoder;
-import org.apache.beam.sdk.coders.protobuf.ProtoCoder;
-import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.Sink.WriteOperation;
 import org.apache.beam.sdk.io.Sink.Writer;
 import org.apache.beam.sdk.options.GcpOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Values;
 import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
 import org.apache.beam.sdk.util.AttemptBoundedExponentialBackOff;
 import org.apache.beam.sdk.util.RetryHttpRequestInitializer;
+import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PDone;
@@ -54,7 +59,6 @@ import com.google.api.client.util.Sleeper;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.MoreObjects;
 import com.google.common.collect.ImmutableList;
-import com.google.common.primitives.Ints;
 import com.google.datastore.v1beta3.CommitRequest;
 import com.google.datastore.v1beta3.Entity;
 import com.google.datastore.v1beta3.EntityResult;
@@ -79,7 +83,6 @@ import org.slf4j.LoggerFactory;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Iterator;
 import java.util.List;
 import java.util.NoSuchElementException;
 
@@ -99,7 +102,8 @@ import javax.annotation.Nullable;
  * To read a {@link PCollection} from a query to Datastore, use {@link 
V1Beta3#read} and
  * its methods {@link V1Beta3.Read#withProjectId} and {@link 
V1Beta3.Read#withQuery} to
  * specify the project to query and the query to read from. You can optionally 
provide a namespace
- * to query within using {@link V1Beta3.Read#withNamespace}.
+ * to query within using {@link V1Beta3.Read#withNamespace}. You could also 
optionally specify
+ * how many splits you want for the query using {@link 
V1Beta3.Read#withNumQuerySplits}.
  *
  * For example:
  *
@@ -168,11 +172,12 @@ public class V1Beta3 {
 
   /**
* Returns an 

[2/2] incubator-beam git commit: Closes #739

[dhalperi]

Closes #739


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bef1a69f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bef1a69f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bef1a69f

Branch: refs/heads/master
Commit: bef1a69f115ffc0cc0e989149f02d1a54dbeedc3
Parents: 8b1e64a 9203013
Author: Dan Halperin 
Authored: Mon Aug 1 22:44:26 2016 -0700
Committer: Dan Halperin 
Committed: Mon Aug 1 22:44:26 2016 -0700

--
 .../beam/sdk/io/gcp/datastore/V1Beta3.java  | 816 +--
 .../beam/sdk/io/gcp/datastore/V1Beta3Test.java  | 466 +--
 2 files changed, 615 insertions(+), 667 deletions(-)
--

[jira] [Updated] (BEAM-515) Add feature logo and incubator logo

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-515:
-
Priority: Critical  (was: Major)

> Add feature logo and incubator logo
> ---
>
> Key: BEAM-515
> URL: https://issues.apache.org/jira/browse/BEAM-515
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: James Malone
>Priority: Critical
>
> Except from: 
> http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E
>  A feather ASF logo would be a nice addition as well. [4]
> http://www.apache.org/foundation/press/kit/#links
> While we're in there, I believe we still need to add the Apache Incubator egg 
> logo. http://incubator.apache.org/images/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-515) Add feature logo and incubator logo

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-515:


 Summary: Add feature logo and incubator logo
 Key: BEAM-515
 URL: https://issues.apache.org/jira/browse/BEAM-515
 Project: Beam
  Issue Type: Bug
  Components: website
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: James Malone


Except from: 
http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E

> Branding wise I think you are missing a few of the
required links [3] including a link back to the Apache homepage.

http://www.apache.org/foundation/marks/pmcs.html#navigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-514) Add all mandatory links

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-514:


 Summary: Add all mandatory links
 Key: BEAM-514
 URL: https://issues.apache.org/jira/browse/BEAM-514
 Project: Beam
  Issue Type: Bug
  Components: website
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: James Malone


Except from: 
http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E

> Branding wise I think you are missing a few of the
required links [3] including a link back to the Apache homepage.

http://www.apache.org/foundation/marks/pmcs.html#navigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-513) Broken link on programming-guide page

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-513:


 Summary: Broken link on programming-guide page
 Key: BEAM-513
 URL: https://issues.apache.org/jira/browse/BEAM-513
 Project: Beam
  Issue Type: Bug
  Components: website
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin


Excerpt from 
http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E:

> this link [2] in Learn menu goes nowhere.

1 . http://beam.incubator.apache.org/releases/
2. http://beam.incubator.apache.org/docs/learn/programming-guide/
3. http://www.apache.org/foundation/marks/pmcs.html#navigation
4. http://www.apache.org/foundation/press/kit/#links
5. http://www.apache.org/dev/release-download-pages.html#closer




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-512) Fill in the contribute/testing section of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-512:
--

 Summary: Fill in the contribute/testing section of the website
 Key: BEAM-512
 URL: https://issues.apache.org/jira/browse/BEAM-512
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-511) Fill in the contribute/technical-vision section of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-511:
--

 Summary: Fill in the contribute/technical-vision section of the 
website
 Key: BEAM-511
 URL: https://issues.apache.org/jira/browse/BEAM-511
 Project: Beam
  Issue Type: Bug
Reporter: Frances Perry


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #29: [BEAM-510] releases: use ASF download ...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam-site/pull/29

[BEAM-510] releases: use ASF download script to balance load on Apache

From:

> All links to the mirrored distribution artifacts must not reference the
> main Apache web site. They should use the standard mechanisms to
> distribute the load between the mirrors.

http://www.apache.org/dev/release-download-pages.html#download-scripts

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam-site beam-510

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/29.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #29


commit 4449337e278ca10c71175b9565d7dd768d3b9ee0
Author: Dan Halperin 
Date:   2016-08-02T04:42:39Z

[BEAM-510] releases: use ASF download script to balance load on Apache

From:

> All links to the mirrored distribution artifacts must not reference the
> main Apache web site. They should use the standard mechanisms to
> distribute the load between the mirrors.

http://www.apache.org/dev/release-download-pages.html#download-scripts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-510) releases: link to source release should use Apache mirror script

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-510:


 Summary: releases: link to source release should use Apache mirror 
script
 Key: BEAM-510
 URL: https://issues.apache.org/jira/browse/BEAM-510
 Project: Beam
  Issue Type: Bug
  Components: website
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin


Except from: 
http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E

> You have a direct link to your last release here [1] rather than a link going 
> via the Apache mirrors script [5]

1 . http://beam.incubator.apache.org/releases/
2. http://beam.incubator.apache.org/docs/learn/programming-guide/
3. http://www.apache.org/foundation/marks/pmcs.html#navigation
4. http://www.apache.org/foundation/press/kit/#links
5. http://www.apache.org/dev/release-download-pages.html#closer



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-509) Fill in the learn/resources portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-509:
--

 Summary: Fill in the learn/resources portion of the website
 Key: BEAM-509
 URL: https://issues.apache.org/jira/browse/BEAM-509
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit

Do a nicer curation of great Beam articles, videos, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-508) Fill in the learn/runners/dataflow portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-508:
--

 Summary: Fill in the learn/runners/dataflow portion of the website
 Key: BEAM-508
 URL: https://issues.apache.org/jira/browse/BEAM-508
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
Should be a landing page for Dataflow-runner-specific content



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-507) Fill in the learn/runners/spark portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-507:
--

 Summary: Fill in the learn/runners/spark portion of the website
 Key: BEAM-507
 URL: https://issues.apache.org/jira/browse/BEAM-507
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
Should be a landing page for Spark-specific information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-506) Fill in the learn/runners/flink portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-506:
--

 Summary: Fill in the learn/runners/flink portion of the website
 Key: BEAM-506
 URL: https://issues.apache.org/jira/browse/BEAM-506
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
Should be a landing page for Flink-specific details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-505) Fill in the learn/runners/direct portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-505:
--

 Summary: Fill in the learn/runners/direct portion of the website
 Key: BEAM-505
 URL: https://issues.apache.org/jira/browse/BEAM-505
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.

Should be a landing page for the Direct runner



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-504) Fill in the learn/sdks/java portion of the website

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-504:
---
Summary: Fill in the learn/sdks/java portion of the website  (was: Fill in 
use/sdks/java portion of the website)

> Fill in the learn/sdks/java portion of the website
> --
>
> Key: BEAM-504
> URL: https://issues.apache.org/jira/browse/BEAM-504
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: James Malone
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Java-SDK-specific content like existing IO 
> connectors, javadoc, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-504) Fill in use/sdks/java portion of the website

[Frances Perry (JIRA)]

Frances Perry created BEAM-504:
--

 Summary: Fill in use/sdks/java portion of the website
 Key: BEAM-504
 URL: https://issues.apache.org/jira/browse/BEAM-504
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As per 
https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.

Should be a landing page for Java-SDK-specific content like existing IO 
connectors, javadoc, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #764: Add ElementByteSizeObserver default constr...

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/764

Add ElementByteSizeObserver default constructor

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This enables constructing an observer without the Counter to simplify
testing and use with other ways of reporting byte size.

Once compatibility issues are resolved, the class will become abstract
with the Counter-specific functionality removed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam 
element-byte-size-observer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/764.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #764


commit 243edfec7e2593dcf8c49541da437a3a6744b5d6
Author: bchambers 
Date:   2016-08-01T17:37:14Z

Add ElementByteSizeObserver default constructor

This enables constructing an observer without the Counter to simplify
testing and use with other ways of reporting byte size.

Once compatibility issues are resolved, the class will become abstract
with the Counter-specific functionality removed.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402815#comment-15402815
 ] 

Aljoscha Krettek commented on BEAM-321:
---

I think we agreed a while back to only put the "fix version" tag once something 
was implemented. So that we don't have to go trough all the issues and remove 
the tag from those that were not actually implemented.

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.2.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-503) FileBasedSource should take a list of files/globs

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-503:
---

Assignee: Chamikara Jayalath  (was: Frances Perry)

> FileBasedSource should take a list of files/globs
> -
>
> Key: BEAM-503
> URL: https://issues.apache.org/jira/browse/BEAM-503
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> This is the Python SDK version of following issue.
> https://issues.apache.org/jira/browse/BEAM-76



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-503) FileBasedSource should take a list of files/globs

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-503:
---

 Summary: FileBasedSource should take a list of files/globs
 Key: BEAM-503
 URL: https://issues.apache.org/jira/browse/BEAM-503
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Frances Perry


This is the Python SDK version of following issue.

https://issues.apache.org/jira/browse/BEAM-76



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-500) Update website layout

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-500:
---
Summary: Update website layout  (was: Update website layou)

> Update website layout
> -
>
> Key: BEAM-500
> URL: https://issues.apache.org/jira/browse/BEAM-500
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Frances Perry
>
> As discussed on dev@, update the website layout to use this:
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-500) Update website layout

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402601#comment-15402601
 ] 

Frances Perry commented on BEAM-500:


(To be clear, this is just the page/navigation structure. The skin / main page 
is covered in BEAM-501.)

> Update website layout
> -
>
> Key: BEAM-500
> URL: https://issues.apache.org/jira/browse/BEAM-500
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Frances Perry
>
> As discussed on dev@, update the website layout to use this:
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-502) Properly handle None/null in json conversions

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-502:
---

 Summary: Properly handle None/null in json conversions
 Key: BEAM-502
 URL: https://issues.apache.org/jira/browse/BEAM-502
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


json_value.py  has to be updated to properly handle JSON to/from Python 'None' 
conversions.

For example, currently writing a dictionary of the form {'aa': 'value', 
'bb':None} using BigQuery sink fails when using DirectPipelineRunner since we 
do not properly handle 'None' values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-500) Update website layout

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402598#comment-15402598
 ] 

Frances Perry commented on BEAM-500:


Devin started this process in 
https://github.com/apache/incubator-beam-site/pull/25

I'll do the next round.

> Update website layout
> -
>
> Key: BEAM-500
> URL: https://issues.apache.org/jira/browse/BEAM-500
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Frances Perry
>
> As discussed on dev@, update the website layout to use this:
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-501) Update website skin

[Frances Perry (JIRA)]

Frances Perry created BEAM-501:
--

 Summary: Update website skin
 Key: BEAM-501
 URL: https://issues.apache.org/jira/browse/BEAM-501
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Frances Perry
Assignee: Jean-Baptiste Onofré


Update the main landing page and website skin as discussed here

https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-500) Update website layou

[Frances Perry (JIRA)]

Frances Perry created BEAM-500:
--

 Summary: Update website layou
 Key: BEAM-500
 URL: https://issues.apache.org/jira/browse/BEAM-500
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Frances Perry
Assignee: Frances Perry


As discussed on dev@, update the website layout to use this:

https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402426#comment-15402426
 ] 

Daniel Halperin commented on BEAM-321:
--

Naive question: is labeling issues with "desired fix version" normal procedure? 
I think the reason we have not done this is because it makes them show up in 
the release notes.

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.2.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402422#comment-15402422
 ] 

Daniel Halperin commented on BEAM-321:
--

Thanks. I think what happened is we accidentally removed on Sunday the fix 
version from this issue (which has been fixed) instead of from 
https://issues.apache.org/jira/browse/BEAM-322 (which has not).

As of this comment, this issue is correctly tagged 0.2.0-incubating and the 
other issue is not.

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.2.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-322) Compare encoded keys in streaming mode

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-322:
-
Affects Version/s: 0.2.0-incubating

> Compare encoded keys in streaming mode
> --
>
> Key: BEAM-322
> URL: https://issues.apache.org/jira/browse/BEAM-322
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-322) Compare encoded keys in streaming mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402355#comment-15402355
 ] 

Aljoscha Krettek commented on BEAM-322:
---

The fix for this is included in this commit of my side-input PR: 
https://github.com/apache/incubator-beam/pull/737/commits/ef9e7c480b55764efd03225109819d5a959c825b#diff-e3189955bcae0a6462c697ba467a303b


> Compare encoded keys in streaming mode
> --
>
> Key: BEAM-322
> URL: https://issues.apache.org/jira/browse/BEAM-322
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402329#comment-15402329
 ] 

Aljoscha Krettek commented on BEAM-321:
---

This was fixed a while ago in master and is part of the 0.2.0-incubating 
release.

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.2.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-321) Hash encoded keys in Flink batch mode

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-321:
--
Fix Version/s: (was: 0.3.0-incubating)
   0.2.0-incubating

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.2.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-315) Flink Runner compares keys unencoded which may produce incorrect results

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-315:

Fix Version/s: 0.3.0-incubating

> Flink Runner compares keys unencoded which may produce incorrect results
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Pawel Szczur
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
> Attachments: CoGroupPipelineStringKey.java, execution.log, 
> execution_split.log, execution_split_sorted.log
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-321) Hash encoded keys in Flink batch mode

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-321:

Fix Version/s: 0.3.0-incubating

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-322) Compare encoded keys in streaming mode

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-322:

Fix Version/s: 0.3.0-incubating

> Compare encoded keys in streaming mode
> --
>
> Key: BEAM-322
> URL: https://issues.apache.org/jira/browse/BEAM-322
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402091#comment-15402091
 ] 

Maximilian Michels commented on BEAM-321:
-

Added the new fix version.

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-321) Hash encoded keys in Flink batch mode

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402084#comment-15402084
 ] 

Maximilian Michels commented on BEAM-321:
-

Since the issue is of pretty high priority, could we add a new fix version for 
the next release?

> Hash encoded keys in Flink batch mode
> -
>
> Key: BEAM-321
> URL: https://issues.apache.org/jira/browse/BEAM-321
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
>
> Right now, hashing of keys happens on the value itself not on the encoded 
> representation. This is at odds with the Beam specification and can lead to 
> incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)