Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2884

2017-03-11 Thread Apache Jenkins Server 
See

[jira] [Updated] (BEAM-1581) JsonIO

2017-03-11 Thread Aviem Zur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1581:

Description: 
A new IO (with source and sink) which will read/write Json files.
Similarly to {{XmlSource}}/{{XmlSink}}, this IO should have a 
{{JsonSource}}/{{JonSink}} which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

  was:
A new IO (with source and sink) which will read/write Json files.
Similar to {{XmlSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}


> JsonIO
> --
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> A new IO (with source and sink) which will read/write Json files.
> Similarly to {{XmlSource}}/{{XmlSink}}, this IO should have a 
> {{JsonSource}}/{{JonSink}} which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1568) Ineffective null check in IsmFormat#structuralValue

2017-03-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906416#comment-15906416
 ] 

ASF GitHub Bot commented on BEAM-1568:
--

GitHub user tedyu opened a pull request:

https://github.com/apache/beam/pull/2227

BEAM-1568 neffective null check in IsmFormat#structuralValue

Add check for record at the beginning of the method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2227.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2227


commit 2326794e59a824dbb72f552034a1c18fae7d11dc
Author: tedyu 
Date:   2017-03-12T03:59:19Z

BEAM-1568 neffective null check in IsmFormat#structuralValue




> Ineffective null check in IsmFormat#structuralValue
> ---
>
> Key: BEAM-1568
> URL: https://issues.apache.org/jira/browse/BEAM-1568
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public Object structuralValue(IsmRecord record) throws Exception {
>   checkState(record.getKeyComponents().size() == 
> keyComponentCoders.size(),
>   "Expected the number of key component coders %s "
>   + "to match the number of key components %s.",
>   keyComponentCoders.size(), record.getKeyComponents());
>   if (record != null && consistentWithEquals()) {
> {code}
> record is de-referenced before the null check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2227: BEAM-1568 neffective null check in IsmFormat#struct...

2017-03-11 Thread tedyu 
GitHub user tedyu opened a pull request:

https://github.com/apache/beam/pull/2227

BEAM-1568 neffective null check in IsmFormat#structuralValue

Add check for record at the beginning of the method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2227.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2227


commit 2326794e59a824dbb72f552034a1c18fae7d11dc
Author: tedyu 
Date:   2017-03-12T03:59:19Z

BEAM-1568 neffective null check in IsmFormat#structuralValue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1568) Ineffective null check in IsmFormat#structuralValue

2017-03-11 Thread Davor Bonaci (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906412#comment-15906412
 ] 

Davor Bonaci commented on BEAM-1568:


[~tedyu], do you want to perhaps submit a fix? Would be much appreciated.

> Ineffective null check in IsmFormat#structuralValue
> ---
>
> Key: BEAM-1568
> URL: https://issues.apache.org/jira/browse/BEAM-1568
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public Object structuralValue(IsmRecord record) throws Exception {
>   checkState(record.getKeyComponents().size() == 
> keyComponentCoders.size(),
>   "Expected the number of key component coders %s "
>   + "to match the number of key components %s.",
>   keyComponentCoders.size(), record.getKeyComponents());
>   if (record != null && consistentWithEquals()) {
> {code}
> record is de-referenced before the null check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1568) Ineffective null check in IsmFormat#structuralValue

2017-03-11 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated BEAM-1568:
-
Description: 
{code}
public Object structuralValue(IsmRecord record) throws Exception {
  checkState(record.getKeyComponents().size() == keyComponentCoders.size(),
  "Expected the number of key component coders %s "
  + "to match the number of key components %s.",
  keyComponentCoders.size(), record.getKeyComponents());

  if (record != null && consistentWithEquals()) {
{code}

record is de-referenced before the null check.

  was:
{code}
public Object structuralValue(IsmRecord record) throws Exception {
  checkState(record.getKeyComponents().size() == keyComponentCoders.size(),
  "Expected the number of key component coders %s "
  + "to match the number of key components %s.",
  keyComponentCoders.size(), record.getKeyComponents());

  if (record != null && consistentWithEquals()) {
{code}
record is de-referenced before the null check.


> Ineffective null check in IsmFormat#structuralValue
> ---
>
> Key: BEAM-1568
> URL: https://issues.apache.org/jira/browse/BEAM-1568
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public Object structuralValue(IsmRecord record) throws Exception {
>   checkState(record.getKeyComponents().size() == 
> keyComponentCoders.size(),
>   "Expected the number of key component coders %s "
>   + "to match the number of key components %s.",
>   keyComponentCoders.size(), record.getKeyComponents());
>   if (record != null && consistentWithEquals()) {
> {code}
> record is de-referenced before the null check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2883

2017-03-11 Thread Apache Jenkins Server 
See

[jira] [Commented] (BEAM-1693) Detect suitable Python & pip executables in Python-SDK

2017-03-11 Thread Tibor Kiss (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906314#comment-15906314
 ] 

Tibor Kiss commented on BEAM-1693:
--

Per [~altay]'s recommendation no shell script should be used to avoid platform 
dependent code.

I'll be proceeding with groovy-maven-plugin to determine the most suitable 
executables (thanks for idea, [~elek]!).

> Detect suitable Python & pip executables in Python-SDK
> --
>
> Key: BEAM-1693
> URL: https://issues.apache.org/jira/browse/BEAM-1693
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>
> Python SDK currently supports Python-2.7 only.
> The Python interpreter & pip definition in pom.xml points to {{python2}} & 
> {{pip2}} respectively. 
> Users with multiple Python interpreters installed might end up having python2 
> and pip2 pointing to their 2.6 installation. (This scenario happens mostly on 
> OS X machines.)
> There is no single, valid name for the executables as different OSes install 
> those binaries in various names:
> - CentOS6/EPEL: pip (python 2.6) & pip2 (python 2.6) & pip2.6 (python 2.6)
> - CentOS7/EPEL: pip (python 2.7) & pip2 (python 2.7) & pip2.7 (python 2.7)
> - Debian7: pip (python 2.7) & pip-2.6 (python 2.6) & pip-2.7 (python 2.7)
> - Debian8: pip (python 2.7) & pip2 (python 2.7)
> - Debian9: pip (python 2.7) & pip2 (python 2.7)
> - Ubuntu1204: pip (python 2.7)
> - Ubuntu1404: pip2 (python 2.7)
> - Ubuntu1604: pip (python 2.7) & pip2 (python 2.7)
> - OS X: pip (python 2.6) & pip2 (python 2.6) & pip2.7 (brew / python 2.7)
> - Windows: pip-2.7 (python.org based installer)
> To overcome this problem the pom.xml should be extended to determine the 
> suitable Python interpreter & pip automatically, in a platform independent 
> way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1693) Detect suitable Python & pip executables in Python-SDK

2017-03-11 Thread Tibor Kiss (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906313#comment-15906313
 ] 

Tibor Kiss commented on BEAM-1693:
--

This ticket was initially part of BEAM-1546.

> Detect suitable Python & pip executables in Python-SDK
> --
>
> Key: BEAM-1693
> URL: https://issues.apache.org/jira/browse/BEAM-1693
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>
> Python SDK currently supports Python-2.7 only.
> The Python interpreter & pip definition in pom.xml points to {{python2}} & 
> {{pip2}} respectively. 
> Users with multiple Python interpreters installed might end up having python2 
> and pip2 pointing to their 2.6 installation. (This scenario happens mostly on 
> OS X machines.)
> There is no single, valid name for the executables as different OSes install 
> those binaries in various names:
> - CentOS6/EPEL: pip (python 2.6) & pip2 (python 2.6) & pip2.6 (python 2.6)
> - CentOS7/EPEL: pip (python 2.7) & pip2 (python 2.7) & pip2.7 (python 2.7)
> - Debian7: pip (python 2.7) & pip-2.6 (python 2.6) & pip-2.7 (python 2.7)
> - Debian8: pip (python 2.7) & pip2 (python 2.7)
> - Debian9: pip (python 2.7) & pip2 (python 2.7)
> - Ubuntu1204: pip (python 2.7)
> - Ubuntu1404: pip2 (python 2.7)
> - Ubuntu1604: pip (python 2.7) & pip2 (python 2.7)
> - OS X: pip (python 2.6) & pip2 (python 2.6) & pip2.7 (brew / python 2.7)
> - Windows: pip-2.7 (python.org based installer)
> To overcome this problem the pom.xml should be extended to determine the 
> suitable Python interpreter & pip automatically, in a platform independent 
> way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1693) Detect suitable Python & pip executables in Python-SDK

2017-03-11 Thread Tibor Kiss (JIRA) 
Tibor Kiss created BEAM-1693:


 Summary: Detect suitable Python & pip executables in Python-SDK
 Key: BEAM-1693
 URL: https://issues.apache.org/jira/browse/BEAM-1693
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Tibor Kiss
Assignee: Tibor Kiss


Python SDK currently supports Python-2.7 only.

The Python interpreter & pip definition in pom.xml points to {{python2}} & 
{{pip2}} respectively. 

Users with multiple Python interpreters installed might end up having python2 
and pip2 pointing to their 2.6 installation. (This scenario happens mostly on 
OS X machines.)

There is no single, valid name for the executables as different OSes install 
those binaries in various names:
- CentOS6/EPEL: pip (python 2.6) & pip2 (python 2.6) & pip2.6 (python 2.6)
- CentOS7/EPEL: pip (python 2.7) & pip2 (python 2.7) & pip2.7 (python 2.7)
- Debian7: pip (python 2.7) & pip-2.6 (python 2.6) & pip-2.7 (python 2.7)
- Debian8: pip (python 2.7) & pip2 (python 2.7)
- Debian9: pip (python 2.7) & pip2 (python 2.7)
- Ubuntu1204: pip (python 2.7)
- Ubuntu1404: pip2 (python 2.7)
- Ubuntu1604: pip (python 2.7) & pip2 (python 2.7)
- OS X: pip (python 2.6) & pip2 (python 2.6) & pip2.7 (brew / python 2.7)
- Windows: pip-2.7 (python.org based installer)

To overcome this problem the pom.xml should be extended to determine the 
suitable Python interpreter & pip automatically, in a platform independent way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2882

2017-03-11 Thread Apache Jenkins Server 
See

[jira] [Closed] (BEAM-789) Review python sdk dependencies

2017-03-11 Thread Ahmet Altay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay closed BEAM-789.

   Resolution: Fixed
Fix Version/s: Not applicable

> Review python sdk dependencies
> --
>
> Key: BEAM-789
> URL: https://issues.apache.org/jira/browse/BEAM-789
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Tibor Kiss
>Priority: Minor
> Fix For: Not applicable
>
>
> Review the existing dependencies for the python sdk. Are they still all 
> required? (e.g. protorpc might not be a required dependency any more.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2881

2017-03-11 Thread Apache Jenkins Server 
See

[jira] [Commented] (BEAM-1581) JsonIO

2017-03-11 Thread Aviem Zur (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906228#comment-15906228
 ] 

Aviem Zur commented on BEAM-1581:
-

Yes, these should definitely be a {{FileBasedSource}}/{{FileBasedSink}}.

I believe the file could either be a JSON array representing the 
{{PCollection}} or a single object with one field which is an array 
representing the {{PCollection}} and other fields for the user to add metadata 
(Good for integrations). Both these options should probably be supported. We 
should definitely hear from [~eljefe6aa] on this.

We have the {{org.apache.beam.sdk.extensions.jackson}} package which has 
{{AsJsons}} and {{ParseJsons}}.
IIUC you're suggesting {{JsonIO}} should not call these directly since the user 
might not want to use Jackson specifically.
How about we have an abstract {{JsonIO}} with a single implementation (right 
now) which will use the Jackson transformations, we can call it {{JacksonIO}}. 
This will allow extensibility and new {{JsonIO}} implementations could be 
introduced in the future.

P.S. {{XmlSource/XmlSink}} do not seem to allow this flexibility and uses JAXB 
directly and does not allow the user to change properties of the Marshaller and 
Unmarshaller.


> JsonIO
> --
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> A new IO (with source and sink) which will read/write Json files.
> Similar to {{XmlSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1551) Allow PAssert#that to take a message

2017-03-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906223#comment-15906223
 ] 

ASF GitHub Bot commented on BEAM-1551:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2109


> Allow PAssert#that to take a message
> 
>
> Key: BEAM-1551
> URL: https://issues.apache.org/jira/browse/BEAM-1551
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Aviem Zur
>
> This permits users to describe their PAsserts, which should show up in the 
> failure message.
> Additionally, the default message of a PAssert failure should contain the 
> name of the input PValue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2109: [BEAM-1551] Allow `PAssert`s to take a message

2017-03-11 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2109


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[5/6] beam git commit: Remove exception suppression from PAssert.SideInputCheckerDoFn

2017-03-11 Thread kenn 
Remove exception suppression from PAssert.SideInputCheckerDoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3669146c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3669146c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3669146c

Branch: refs/heads/master
Commit: 3669146c95e265c0fbdece7d04f6b787cdac
Parents: 59fd45b
Author: Aviem Zur 
Authored: Fri Mar 10 23:15:15 2017 +0200
Committer: Aviem Zur 
Committed: Fri Mar 10 23:15:15 2017 +0200

--
 .../main/java/org/apache/beam/sdk/testing/PAssert.java  | 12 ++--
 1 file changed, 2 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3669146c/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index d88c4d6..2596335 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -40,7 +40,6 @@ import org.apache.beam.sdk.coders.IterableCoder;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.MapCoder;
 import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.options.StreamingOptions;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.runners.TransformHierarchy.Node;
 import org.apache.beam.sdk.transforms.Aggregator;
@@ -1102,15 +1101,8 @@ public class PAssert {
 
 @ProcessElement
 public void processElement(ProcessContext c) {
-  try {
-ActualT actualContents = c.sideInput(actual);
-doChecks(actualContents, checkerFn, success, failure);
-  } catch (Throwable t) {
-// Suppress exception in streaming
-if (!c.getPipelineOptions().as(StreamingOptions.class).isStreaming()) {
-  throw t;
-}
-  }
+  ActualT actualContents = c.sideInput(actual);
+  doChecks(actualContents, checkerFn, success, failure);
 }
   }

[2/6] beam git commit: [BEAM-1551] Allow `PAssert`s to take a message

2017-03-11 Thread kenn 
[BEAM-1551] Allow `PAssert`s to take a message


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e3cafb42
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e3cafb42
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e3cafb42

Branch: refs/heads/master
Commit: e3cafb42cdea1a35e067704e3a8f1277549aff67
Parents: 818fc94
Author: Aviem Zur 
Authored: Sun Feb 26 19:42:13 2017 +0200
Committer: Aviem Zur 
Committed: Fri Mar 10 23:13:38 2017 +0200

--
 .../org/apache/beam/sdk/testing/PAssert.java| 168 ++-
 .../apache/beam/sdk/testing/PAssertTest.java|   3 +-
 2 files changed, 130 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e3cafb42/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index a6fb232e..1faa024 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -279,7 +279,14 @@ public class PAssert {
* Constructs an {@link IterableAssert} for the elements of the provided 
{@link PCollection}.
*/
   public static  IterableAssert that(PCollection actual) {
-return new PCollectionContentsAssert<>(actual);
+return that(actual.getName(), actual);
+  }
+
+  /**
+   * Constructs an {@link IterableAssert} for the elements of the provided 
{@link PCollection}.
+   */
+  public static  IterableAssert that(String reason, PCollection 
actual) {
+return new PCollectionContentsAssert<>(reason, actual);
   }
 
   /**
@@ -288,6 +295,15 @@ public class PAssert {
*/
   public static  IterableAssert thatSingletonIterable(
   PCollection> actual) {
+return thatSingletonIterable(actual.getName(), actual);
+  }
+
+  /**
+   * Constructs an {@link IterableAssert} for the value of the provided {@link 
PCollection} which
+   * must contain a single {@code Iterable} value.
+   */
+  public static  IterableAssert thatSingletonIterable(
+  String reason, PCollection> actual) {
 
 try {
 } catch (NoSuchElementException | IllegalArgumentException exc) {
@@ -300,7 +316,7 @@ public class PAssert {
 @SuppressWarnings("unchecked") // Safe covariant cast
 PCollection actualIterables = (PCollection) 
actual;
 
-return new PCollectionSingletonIterableAssert<>(actualIterables);
+return new PCollectionSingletonIterableAssert<>(reason, actualIterables);
   }
 
   /**
@@ -308,7 +324,15 @@ public class PAssert {
* {@code PCollection PCollection}, which must be a singleton.
*/
   public static  SingletonAssert thatSingleton(PCollection actual) {
-return new PCollectionViewAssert<>(actual, View.asSingleton(), 
actual.getCoder());
+return thatSingleton(actual.getName(), actual);
+  }
+
+  /**
+   * Constructs a {@link SingletonAssert} for the value of the provided
+   * {@code PCollection PCollection}, which must be a singleton.
+   */
+  public static  SingletonAssert thatSingleton(String reason, 
PCollection actual) {
+return new PCollectionViewAssert<>(actual, View.asSingleton(), 
actual.getCoder(), reason);
   }
 
   /**
@@ -319,12 +343,24 @@ public class PAssert {
*/
   public static  SingletonAssert> thatMultimap(
   PCollection> actual) {
+return thatMultimap(actual.getName(), actual);
+  }
+
+  /**
+   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection}.
+   *
+   * Note that the actual value must be coded by a {@link KvCoder}, not 
just any
+   * {@code Coder}.
+   */
+  public static  SingletonAssert> thatMultimap(
+  String reason, PCollection> actual) {
 @SuppressWarnings("unchecked")
 KvCoder kvCoder = (KvCoder) actual.getCoder();
 return new PCollectionViewAssert<>(
 actual,
 View.asMultimap(),
-MapCoder.of(kvCoder.getKeyCoder(), 
IterableCoder.of(kvCoder.getValueCoder(;
+MapCoder.of(kvCoder.getKeyCoder(), 
IterableCoder.of(kvCoder.getValueCoder())),
+reason);
   }
 
   /**
@@ -335,10 +371,23 @@ public class PAssert {
* {@code Coder}.
*/
   public static  SingletonAssert> thatMap(PCollection> actual) {
+return thatMap(actual.getName(), actual);
+  }
+
+  /**
+   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection}, which
+   * must have at most one value per key.
+   *
+   * Note that the actual value must be coded

[3/6] beam git commit: Added a test of default PAssert failure reason

2017-03-11 Thread kenn 
Added a test of default PAssert failure reason


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2485a4cc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2485a4cc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2485a4cc

Branch: refs/heads/master
Commit: 2485a4ccce18784abf520792ea36b7d3d6efc394
Parents: 0d08d2a
Author: Aviem Zur 
Authored: Wed Mar 1 08:10:03 2017 +0200
Committer: Aviem Zur 
Committed: Fri Mar 10 23:13:38 2017 +0200

--
 .../apache/beam/sdk/testing/PAssertTest.java| 21 ++--
 1 file changed, 19 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2485a4cc/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
index f50adf4..9bdb1b5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
@@ -393,8 +393,25 @@ public class PAssertTest implements Serializable {
 
 Throwable thrown = runExpectingAssertionFailure(pipeline);
 
-assertThat(thrown.getMessage(), containsString("Vals should have been 
empty"));
-assertThat(thrown.getMessage(), containsString("Expected: iterable over [] 
in any order"));
+String message = thrown.getMessage();
+
+assertThat(message, containsString("Vals should have been empty"));
+assertThat(message, containsString("Expected: iterable over [] in any 
order"));
+  }
+
+  @Test
+  @Category(RunnableOnService.class)
+  public void testEmptyFalseDefaultReasonString() throws Exception {
+PCollection vals = pipeline.apply(CountingInput.upTo(5L));
+PAssert.that(vals).empty();
+
+Throwable thrown = runExpectingAssertionFailure(pipeline);
+
+String message = thrown.getMessage();
+
+assertThat(message,
+
containsString("CountingInput.BoundedCountingInput/Read(BoundedCountingSource).out"));
+assertThat(message, containsString("Expected: iterable over [] in any 
order"));
   }
 
   private static Throwable runExpectingAssertionFailure(Pipeline pipeline) {

[1/6] beam git commit: Added assertion failure tests for `PAssert#thatSingleton`

2017-03-11 Thread kenn 
Repository: beam
Updated Branches:
  refs/heads/master 7c78480cf -> d16715309


Added assertion failure tests for `PAssert#thatSingleton`


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59fd45b8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59fd45b8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59fd45b8

Branch: refs/heads/master
Commit: 59fd45b8104c7c19c65aa0df31fb9312ff82c650
Parents: 2485a4c
Author: Aviem Zur 
Authored: Wed Mar 1 08:31:57 2017 +0200
Committer: Aviem Zur 
Committed: Fri Mar 10 23:13:38 2017 +0200

--
 .../apache/beam/sdk/testing/PAssertTest.java| 36 
 1 file changed, 36 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/59fd45b8/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
index 9bdb1b5..1603db5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
@@ -297,6 +297,42 @@ public class PAssertTest implements Serializable {
   }
 
   /**
+   * Test that we throw an error for false assertion on singleton.
+   */
+  @Test
+  @Category(RunnableOnService.class)
+  public void testPAssertEqualsSingletonFalse() throws Exception {
+PCollection pcollection = pipeline.apply(Create.of(42));
+PAssert.thatSingleton("The value was not equal to 44", 
pcollection).isEqualTo(44);
+
+Throwable thrown = runExpectingAssertionFailure(pipeline);
+
+String message = thrown.getMessage();
+
+assertThat(message, containsString("The value was not equal to 44"));
+assertThat(message, containsString("Expected: <44>"));
+assertThat(message, containsString("but: was <42>"));
+  }
+
+  /**
+   * Test that we throw an error for false assertion on singleton.
+   */
+  @Test
+  @Category(RunnableOnService.class)
+  public void testPAssertEqualsSingletonFalseDefaultReasonString() throws 
Exception {
+PCollection pcollection = pipeline.apply(Create.of(42));
+PAssert.thatSingleton(pcollection).isEqualTo(44);
+
+Throwable thrown = runExpectingAssertionFailure(pipeline);
+
+String message = thrown.getMessage();
+
+assertThat(message, 
containsString("Create.Values/Read(CreateSource).out"));
+assertThat(message, containsString("Expected: <44>"));
+assertThat(message, containsString("but: was <42>"));
+  }
+
+  /**
* Tests that {@code containsInAnyOrder} is actually order-independent.
*/
   @Test

[6/6] beam git commit: This closes #2109: Allow `PAssert`s to take a message

2017-03-11 Thread kenn 
This closes #2109: Allow `PAssert`s to take a message

  Remove exception suppression from PAssert.SideInputCheckerDoFn
  Added assertion failure tests for `PAssert#thatSingleton`
  Added a test of default PAssert failure reason
  Javadoc changes
  [BEAM-1551] Allow `PAssert`s to take a message


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d1671530
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d1671530
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d1671530

Branch: refs/heads/master
Commit: d1671530951bd67956f6d022b523053c12a04df3
Parents: 7c78480 3669146
Author: Kenneth Knowles 
Authored: Sat Mar 11 07:04:30 2017 -0800
Committer: Kenneth Knowles 
Committed: Sat Mar 11 07:04:30 2017 -0800

--
 .../org/apache/beam/sdk/testing/PAssert.java| 183 ++-
 .../apache/beam/sdk/testing/PAssertTest.java|  56 +-
 2 files changed, 188 insertions(+), 51 deletions(-)
--

[4/6] beam git commit: Javadoc changes

2017-03-11 Thread kenn 
Javadoc changes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d08d2a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d08d2a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d08d2a4

Branch: refs/heads/master
Commit: 0d08d2a4561b286711ea4322f8d53ce5ad11e89f
Parents: e3cafb4
Author: Aviem Zur 
Authored: Wed Mar 1 07:51:05 2017 +0200
Committer: Aviem Zur 
Committed: Fri Mar 10 23:13:38 2017 +0200

--
 .../java/org/apache/beam/sdk/testing/PAssert.java  | 17 ++---
 1 file changed, 10 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0d08d2a4/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index 1faa024..d88c4d6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -283,7 +283,8 @@ public class PAssert {
   }
 
   /**
-   * Constructs an {@link IterableAssert} for the elements of the provided 
{@link PCollection}.
+   * Constructs an {@link IterableAssert} for the elements of the provided 
{@link PCollection}
+   * with the specified reason.
*/
   public static  IterableAssert that(String reason, PCollection 
actual) {
 return new PCollectionContentsAssert<>(reason, actual);
@@ -299,8 +300,8 @@ public class PAssert {
   }
 
   /**
-   * Constructs an {@link IterableAssert} for the value of the provided {@link 
PCollection} which
-   * must contain a single {@code Iterable} value.
+   * Constructs an {@link IterableAssert} for the value of the provided {@link 
PCollection } with
+   * the specified reason. The provided PCollection must contain a single 
{@code Iterable} value.
*/
   public static  IterableAssert thatSingletonIterable(
   String reason, PCollection> actual) {
@@ -329,7 +330,8 @@ public class PAssert {
 
   /**
* Constructs a {@link SingletonAssert} for the value of the provided
-   * {@code PCollection PCollection}, which must be a singleton.
+   * {@code PCollection PCollection} with the specified reason. The 
provided PCollection must be
+   * a singleton.
*/
   public static  SingletonAssert thatSingleton(String reason, 
PCollection actual) {
 return new PCollectionViewAssert<>(actual, View.asSingleton(), 
actual.getCoder(), reason);
@@ -347,7 +349,8 @@ public class PAssert {
   }
 
   /**
-   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection}.
+   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection} with the
+   * specified reason.
*
* Note that the actual value must be coded by a {@link KvCoder}, not 
just any
* {@code Coder}.
@@ -375,8 +378,8 @@ public class PAssert {
   }
 
   /**
-   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection}, which
-   * must have at most one value per key.
+   * Constructs a {@link SingletonAssert} for the value of the provided {@link 
PCollection} with
+   * the specified reason. The {@link PCollection} must have at most one value 
per key.
*
* Note that the actual value must be coded by a {@link KvCoder}, not 
just any
* {@code Coder}.

[jira] [Updated] (BEAM-1692) Convert to Codahale Metric interfaces where possible in WithMetricsSupport MetricRegistry

2017-03-11 Thread Aviem Zur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1692:

Description: 
Currently in Spark runner all Beam metrics are reported by Codahale metrics as 
Codahale {{Gauge}} metrics.
Some of the metrics are formatted to look like their counterparts in Codahale's 
{{GraphiteReporter}} but this is a hack and should be avoided if possible.

Instead, convert Beam metrics to their Codahale counterparts in 
{{WithMetricsSupport}} getters where possible.

Since Beam metrics are not 100% compatible with Codahale metrics, consider 
where to make tradeoffs and stay with a simple {{Gauge}}.

Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}

  was:
Currently in Spark runner all Beam metrics are reported by Codahale metrics as 
Codahale {{Gauge}}s.
Some of the metrics are formatted to look like their counterparts in Codahale's 
{{GraphiteReporter}} but this is a hack and should be avoided if possible.

Instead, convert Beam metrics to their Codahale counterparts in 
{{WithMetricsSupport}} getters where possible.

Since Beam metrics are not 100% compatible with Codahale metrics, consider 
where to make tradeoffs and stay with a simple {{Gauge}}.

Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}


> Convert to Codahale Metric interfaces where possible in WithMetricsSupport 
> MetricRegistry
> -
>
> Key: BEAM-1692
> URL: https://issues.apache.org/jira/browse/BEAM-1692
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Minor
>
> Currently in Spark runner all Beam metrics are reported by Codahale metrics 
> as Codahale {{Gauge}} metrics.
> Some of the metrics are formatted to look like their counterparts in 
> Codahale's {{GraphiteReporter}} but this is a hack and should be avoided if 
> possible.
> Instead, convert Beam metrics to their Codahale counterparts in 
> {{WithMetricsSupport}} getters where possible.
> Since Beam metrics are not 100% compatible with Codahale metrics, consider 
> where to make tradeoffs and stay with a simple {{Gauge}}.
> Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1692) Convert to Codahale Metric interfaces where possible in WithMetricsSupport MetricRegistry

2017-03-11 Thread Aviem Zur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1692:

Description: 
Currently in Spark runner all Beam metrics are reported by Codahale metrics as 
Codahale {{Gauge}}s.
Some of the metrics are formatted to look like their counterparts in Codahale's 
{{GraphiteReporter}} but this is a hack and should be avoided if possible.

Instead, convert Beam metrics to their Codahale counterparts in 
{{WithMetricsSupport}} getters where possible.

Since Beam metrics are not 100% compatible with Codahale metrics, consider 
where to make tradeoffs and stay with a simple {{Gauge}}.

Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}

  was:
Currently in Spark runner all Beam metrics are reported by Codahale metrics as 
Codahale {{Gauge}}s.
Some of the metrics are formatted to look like their counterparts in Codahale's 
{{GraphiteReporter}} but this is a hack and should be avoided if possible.

Instead, convert Beam metrics to their Codahale counterparts in 
{{WithMetricsSupport}} getters where possible.

Since Beam metrics are not 100% compatible with Codahale metrics, consider 
where to make tradeoffs and stay with a simple {{Gauge}}.


> Convert to Codahale Metric interfaces where possible in WithMetricsSupport 
> MetricRegistry
> -
>
> Key: BEAM-1692
> URL: https://issues.apache.org/jira/browse/BEAM-1692
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Minor
>
> Currently in Spark runner all Beam metrics are reported by Codahale metrics 
> as Codahale {{Gauge}}s.
> Some of the metrics are formatted to look like their counterparts in 
> Codahale's {{GraphiteReporter}} but this is a hack and should be avoided if 
> possible.
> Instead, convert Beam metrics to their Codahale counterparts in 
> {{WithMetricsSupport}} getters where possible.
> Since Beam metrics are not 100% compatible with Codahale metrics, consider 
> where to make tradeoffs and stay with a simple {{Gauge}}.
> Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Apex #739

2017-03-11 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2880

2017-03-11 Thread Apache Jenkins Server 
See