[jira] [Updated] (BEAM-1425) Window should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1425:
--
Labels: backward-incompatible starter  (was: backward-incompatible)

> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1353) Beam should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1353:
--
Labels: backward-incompatible easy starter  (was: backward-incompatible)

> Beam should comply with PTransform style guide
> --
>
> Key: BEAM-1353
> URL: https://issues.apache.org/jira/browse/BEAM-1353
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core, sdk-java-extensions, sdk-java-gcp, sdk-py
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> https://beam.apache.org/contribute/ptransform-style-guide/ is a collection of 
> best practices for developing PTransform's.
> Some of these practices shaped up relatively recently, and some Beam 
> transforms violate them, introducing inconsistency into Beam APIs and 
> tempting new library writers and users into mimicking code that's best not 
> mimicked.
> This is an umbrella bug for cleanups bringing individual Beam transforms in 
> accordance with the style guide.
> Please file 1 bug per transform that needs to be fixed (e.g. "Bring KafkaIO 
> in accordance with style guide"); in the bug, describe which aspects need to 
> be fixed (e.g. "should use AutoValue; should explicitly specify type 
> parameters; should not use Coders for serialization") and link it with the 
> current bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1342) Rectify any egregious violations of the IO authoring guide in our IOs

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1342:
--
Fix Version/s: First stable release

> Rectify any egregious violations of the IO authoring guide in our IOs
> -
>
> Key: BEAM-1342
> URL: https://issues.apache.org/jira/browse/BEAM-1342
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions, sdk-java-gcp
>Reporter: Kenneth Knowles
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-73) IO design pattern: Decouple Parsers and Coders

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-73?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-73:

Fix Version/s: First stable release

> IO design pattern: Decouple Parsers and Coders
> --
>
> Key: BEAM-73
> URL: https://issues.apache.org/jira/browse/BEAM-73
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Many Sources can be thought of as providing a byte[] payload -- e.g. TextIO 
> bytes between newlines, or PubSubIO messages. Therefore, we originally 
> suggested a Coder as the thing to use to decode these byte[] into T (what 
> I'll call Parsing).
> Consider the case of a text file of integers.
> 123\n
> 456\n
> ...
> We want a PCollection out, so we can use TextualIntegerCoder with 
> TextIO.Read. However, that Coder will get propagated as the default coder for 
> that PCollection (and may be used in downstream DoFns). This seem bad as, 
> once the data is parsed, we probably want to use VarIntCoder or another Coder 
> that is more CPU- and Space-efficient.
> Another design pattern is
> TextIO.Read() -> MapElements (lambda s : 
> Integer.parseInt(s))
> This has better behavior, but now we go from byte[] to String to Integer 
> rather than directly from byte[] to Integer.
> The solution seems to be to explicitly add Parser and Coder abstractions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1353) Beam should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1353:
--
Fix Version/s: First stable release

> Beam should comply with PTransform style guide
> --
>
> Key: BEAM-1353
> URL: https://issues.apache.org/jira/browse/BEAM-1353
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core, sdk-java-extensions, sdk-java-gcp, sdk-py
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> https://beam.apache.org/contribute/ptransform-style-guide/ is a collection of 
> best practices for developing PTransform's.
> Some of these practices shaped up relatively recently, and some Beam 
> transforms violate them, introducing inconsistency into Beam APIs and 
> tempting new library writers and users into mimicking code that's best not 
> mimicked.
> This is an umbrella bug for cleanups bringing individual Beam transforms in 
> accordance with the style guide.
> Please file 1 bug per transform that needs to be fixed (e.g. "Bring KafkaIO 
> in accordance with style guide"); in the bug, describe which aspects need to 
> be fixed (e.g. "should use AutoValue; should explicitly specify type 
> parameters; should not use Coders for serialization") and link it with the 
> current bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1418) MapElements and FlatMapElements should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1418:
--
Fix Version/s: First stable release

> MapElements and FlatMapElements should comply with PTransform style guide
> -
>
> Key: BEAM-1418
> URL: https://issues.apache.org/jira/browse/BEAM-1418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Type structure of these classes can be improved by slightly changing the API:
> FlatMapElements.into(TypeDescriptor).via(as usual).
> Likewise for MapElements. This allows getting rid of the awkward 
> MissingOutputTypeDescriptor intermediate class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1415) PubsubIO should comply with PTransfrom style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1415:
--
Fix Version/s: First stable release

> PubsubIO should comply with PTransfrom style guide
> --
>
> Key: BEAM-1415
> URL: https://issues.apache.org/jira/browse/BEAM-1415
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename builder methods such as .subscription(), .topic() etc. to 
> .withSubscription, .withTopic()
> - Replace use of Coder from the API (.withCoder()) with a SerializableFunction
> - Rename .withAttributes() to something else, because it sounds like this is 
> a function that sets attributes.
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1415) PubsubIO should comply with PTransfrom style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1415:
--
Labels: backward-incompatible easy starter  (was: backward-incompatible)

> PubsubIO should comply with PTransfrom style guide
> --
>
> Key: BEAM-1415
> URL: https://issues.apache.org/jira/browse/BEAM-1415
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename builder methods such as .subscription(), .topic() etc. to 
> .withSubscription, .withTopic()
> - Replace use of Coder from the API (.withCoder()) with a SerializableFunction
> - Rename .withAttributes() to something else, because it sounds like this is 
> a function that sets attributes.
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-184) Using Merging Windows and/or Triggers without a downstream aggregation should fail

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903673#comment-15903673
 ] 

Kenneth Knowles commented on BEAM-184:
--

I think with pipeline surgery in Java we are at a point where we can do this 
sort of validation effectively across runners. I will put it on the burn-down 
and choose an arbitrary owner, with the understanding that we have bigger 
priorities.

> Using Merging Windows and/or Triggers without a downstream aggregation should 
> fail
> --
>
> Key: BEAM-184
> URL: https://issues.apache.org/jira/browse/BEAM-184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Both merging windows (such as sessions) and triggering only actually happen 
> at an aggregation (GroupByKey). We should produce errors in any of these 
> cases:
> 1. Merging window used without a downstream GroupByKey
> 2. Triggers used without a downstream GroupByKey
> 3. Window inspected after inserting a merging window and before the GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-184) Using Merging Windows and/or Triggers without a downstream aggregation should fail

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-184:


Assignee: Thomas Groh

> Using Merging Windows and/or Triggers without a downstream aggregation should 
> fail
> --
>
> Key: BEAM-184
> URL: https://issues.apache.org/jira/browse/BEAM-184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Both merging windows (such as sessions) and triggering only actually happen 
> at an aggregation (GroupByKey). We should produce errors in any of these 
> cases:
> 1. Merging window used without a downstream GroupByKey
> 2. Triggers used without a downstream GroupByKey
> 3. Window inspected after inserting a merging window and before the GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-170) Session windows should not be identified by their bounds

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-170:
-
Fix Version/s: First stable release

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Ilya Ganelin
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-184) Using Merging Windows and/or Triggers without a downstream aggregation should fail

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-184:
-
Fix Version/s: First stable release

> Using Merging Windows and/or Triggers without a downstream aggregation should 
> fail
> --
>
> Key: BEAM-184
> URL: https://issues.apache.org/jira/browse/BEAM-184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Both merging windows (such as sessions) and triggering only actually happen 
> at an aggregation (GroupByKey). We should produce errors in any of these 
> cases:
> 1. Merging window used without a downstream GroupByKey
> 2. Triggers used without a downstream GroupByKey
> 3. Window inspected after inserting a merging window and before the GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

svn commit: r18644 - in /dev/beam/0.6.0: apache-beam-0.6.0.tar.gz.asc apache-beam-0.6.0.tar.gz.md5 apache-beam-0.6.0.tar.gz.sha1

[altay]

Author: altay
Date: Thu Mar  9 19:21:03 2017
New Revision: 18644

Log:
Add signatures/hashes for python tarball.


Added:
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.asc
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.md5
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.sha1

Added: dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.asc
==
--- dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.asc (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.asc Thu Mar  9 19:21:03 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJYwapWAAoJENYucUFglvoA+jsQAJcG/Xt9pglc1IFIKUX3VLmU
+jt1WrdD/15zlAQ8tVGLor9vhfF0bO4cR+kCH3yUN36b+oB5Wg74UQAOcUCtIK9mS
++SNSj4XUhFBFH60d1RxfHP7OfqSXBRqyI+QPJB+Ew3iOknq2scbdbgtYALgfwBIm
+G1ct6jsE0/HhR/KUsEo283WRKRCvWi7y9bv6rX97Ul6r018cuesJRyUSqu/B5BXf
++3Y2N2mws2D7EexjUMgABuadYQP0N+yxcy8m5muvYu8K+NdN2bXuqgI8ZGwMHSnB
+UXtbzgG3AwpOm/1gsVSpB9KaWu81KIYLKnj98OYCj3CBzIcMtA+zdUHW5Bq7gXTe
+pdk9Llf2p7osGpHAdLcMI5tQah00R08x1pfeQ/oCTajyv+9HIiDBxe5e4WmLIoEg
+v+usDjfy4QvKJmuJawyOSBz7tKqSecZJWzKPsHs0v7ZZEPT/rcOOtjDSAdf1QmMi
++cHFV0i6aZQGAotpihAoxR56bmKW/K9NSBFWi4IdrvK4h16uAHlxf86KyPylles5
+hroK7kgD91ahT+47hKoeVkbBwTE/NBobv0esUWS8dqcEQ7a3Fh26rjXE10l0vkgA
+MUPnIiPqtsgD3PYzJ+jsun7l0LrIBaSocSWuguIKPwYSJtADDokq8juu5EtsagI4
+bpxxOfwt3/+PMXDM6I1L
+=wBEA
+-END PGP SIGNATURE-

Added: dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.md5
==
--- dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.md5 (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.md5 Thu Mar  9 19:21:03 2017
@@ -0,0 +1 @@
+266034c0a12e65e7468331841b03d826  apache-beam-0.6.0.tar.gz

Added: dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.sha1
==
--- dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.sha1 (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.sha1 Thu Mar  9 19:21:03 2017
@@ -0,0 +1 @@
+c192990ef1eebf71a7a25337c62362a45548fe34  apache-beam-0.6.0.tar.gz

[jira] [Updated] (BEAM-775) Remove Aggregators from the Java SDK

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-775:
-
Fix Version/s: First stable release

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
>  Labels: backward-incompatible
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-885) Move PipelineOptions from Pipeline.create() to Pipeline.run()

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-885:
-
Fix Version/s: First stable release

> Move PipelineOptions from Pipeline.create() to Pipeline.run()
> -
>
> Key: BEAM-885
> URL: https://issues.apache.org/jira/browse/BEAM-885
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The specification of a Pipeline should be independent of its PipelineOptions. 
> This delays specification of the options, including choices like Pipeline 
> Runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1327) Replace OutputTimeFn with enum

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1327:
--
Fix Version/s: First stable release

> Replace OutputTimeFn with enum
> --
>
> Key: BEAM-1327
> URL: https://issues.apache.org/jira/browse/BEAM-1327
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The class {{OutputTimeFn}} is overkill for a Fn API crossing. There are only 
> three sensible values known: MIN, MAX, EOW. The interface is right for 
> implementing these, but the full class is left over from the days when there 
> was little cost to shipping new kinds of fns. An enum is concise.
> This can be done "mostly" backwards compatibly with legacy adapters in place, 
> but might be less confusing without them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1345) Mark @Experimental pieces of the SDK that have been added but are not ready to be frozen

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1345:
--
Fix Version/s: First stable release

> Mark @Experimental pieces of the SDK that have been added but are not ready 
> to be frozen
> 
>
> Key: BEAM-1345
> URL: https://issues.apache.org/jira/browse/BEAM-1345
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions, sdk-java-gcp
>Reporter: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> A blanket JIRA to ensure that before a stable release we make sure to mark 
> those pieces that would be unwise to freeze yet and consider how best to 
> communicate this to users, who may just autocomplete those features in their 
> IDE anyhow. (conversely, put infrastructure in place to enforce freezing of 
> the rest)
> Not technically "backwards incompatible" with pre-Beam Dataflow SDKs, but 
> certainly needs to be on the burndown for the first stable release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1340:
--
Fix Version/s: First stable release

> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1247) Session state should not be lost when discardingFiredPanes

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1247:
--
Fix Version/s: First stable release

> Session state should not be lost when discardingFiredPanes
> --
>
> Key: BEAM-1247
> URL: https://issues.apache.org/jira/browse/BEAM-1247
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Today when {{discardingFiredPanes}} the entirety of state is cleared, 
> including the state of evolving sessions. This means that with multiple 
> triggerings a single session shows up as multiple. This also stymies 
> downstream stateful computations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-846) Decouple side input window mapping from WindowFn

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-846:
-
Fix Version/s: First stable release

> Decouple side input window mapping from WindowFn
> 
>
> Key: BEAM-846
> URL: https://issues.apache.org/jira/browse/BEAM-846
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Currently the main WindowFn provides as getSideInputWindow method. Instead, 
> this mapping should be specified per-side-input (thought the default mapping 
> would remain the same). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-32) Consider not emitting empty ON_TIME pane unless requested

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-32:

Fix Version/s: First stable release

> Consider not emitting empty ON_TIME pane unless requested
> -
>
> Key: BEAM-32
> URL: https://issues.apache.org/jira/browse/BEAM-32
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Triggers, Windowing, backward-incompatible
> Fix For: First stable release
>
>
> Today, the ReduceFnRunner sets a timers and emits an empty ON_TIME pane as 
> long as the trigger allows it. This could be controlled in a manner analogous 
> to the empty final pane at window expiration (also owned by the 
> ReduceFnRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1521) Pass element (and/or generic context) for choosing a table in BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1521:
--
Labels: backward-incompatible  (was: backwards-incompatible)

> Pass element (and/or generic context) for choosing a table in BigQueryIO
> 
>
> Key: BEAM-1521
> URL: https://issues.apache.org/jira/browse/BEAM-1521
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: backward-incompatible
>
> It would be easy and helpful to also pass the element when choosing a table 
> for a BigQuery write. And perhaps future-proof this by a Context parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #175: Documentation changes for 0.6.0 release

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/175

Documentation changes for 0.6.0 release

R: @davorbonaci @sb2nov 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/175.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #175


commit d6ca2944e2c127fe318ac7884140c372e61bd4f1
Author: Ahmet Altay 
Date:   2017-03-09T18:56:40Z

Documentation changes for 0.6.0 release.

commit 178cfc0befb95368c65eb5501588a1fff086e896
Author: Ahmet Altay 
Date:   2017-03-09T18:58:25Z

One time changes for enabling pydocs on the website.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #12

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-1184] Add integration tests to ElasticsearchIO

[amitsela33] Test runner to stop on EOT watermark, or timeout.

[markliu] Add JDK 1.8 version

--
[...truncated 476.31 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.301 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.868 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.092 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running 

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Summary: SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8 on Jenkins  
(was: SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8)

> SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8 on Jenkins
> --
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Description: 
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Any ideas?

  was:
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Any ideas?


> SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8
> ---
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.8 (latest),beam #12

[Apache Jenkins Server]

See 


--
[...truncated 473.90 KB...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.072 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.097 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.905 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running org.apache.beam.sdk.util.FluentBackoffTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.FluentBackoffTest
Running org.apache.beam.sdk.util.IntervalBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 

[jira] [Created] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8

[Mark Liu (JIRA)]

Mark Liu created BEAM-1676:
--

 Summary: SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8
 Key: BEAM-1676
 URL: https://issues.apache.org/jira/browse/BEAM-1676
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Mark Liu
Assignee: Davor Bonaci


After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1674) Flink user state GC depends on order of timer firing

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903587#comment-15903587
 ] 

Kenneth Knowles commented on BEAM-1674:
---

The user needs to get a timer callback late enough that they know they are 
doing a final flush, so their timer has to be for GC time.

The actual GC does not have to happen immediately, as long as incoming elements 
and processing time timers are discarded so the window is inactive. So you can 
set the actual GC timer slightly later (1ms) if you have some guarantee that 
all eligible timers are delivered and they are delivered in timestamp order.

In my implementations I simply have completely separate system timer internals 
and user timer internals in disjoint namespaces, and I process all the user 
timers first. It looks like this is compatible with {{StatefulDoFnRunner}} 
since all that logic would happen in the code that builds the {{CleanupTimer}} 
/ {{TimerInternalsCleanupTimer}}.

> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1872

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903563#comment-15903563
 ] 

Davor Bonaci commented on BEAM-664:
---

[~hadarhg] / [~frances], what's the accurate status of this task and its 
sub-tasks? Perhaps this is done?

> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1650) Add IntelliJ code style xml to the project repo and website

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903560#comment-15903560
 ] 

Aviem Zur commented on BEAM-1650:
-

We might be able to leverage https://github.com/revelc/formatter-maven-plugin 
(If we change to Eclipse code style xml, which could be a good idea since it 
would work in both Eclipse and IntelliJ) or 
https://confluence.jetbrains.com/display/IDEADEV/Command-Line+Source+Code+Formatter
We can use one of these to format a test class before checkstyle runs so if the 
formatting does not match checkstyle it will fail.

> Add IntelliJ code style xml to the project repo and website 
> 
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa could benefit from having a code 
> style xml file which matches the checkstyle enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IntelliJ setup section: 
> https://beam.apache.org/contribute/contribution-guide/#intellij



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1401) Sinks in Beam should supported windowed unbounded PCollections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903562#comment-15903562
 ] 

ASF GitHub Bot commented on BEAM-1401:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2212

[BEAM-1401] Add windowing support to FileBasedSink

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Reuven is on vacation so this takes over #1926 with rebase and squash.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam windowed-io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2212


commit d9649085197a47df744cf9d748a47d429e47aa42
Author: Reuven Lax 
Date:   2017-03-09T17:45:35Z

Add windowing support to FileBasedSink




> Sinks in Beam should supported windowed unbounded PCollections
> --
>
> Key: BEAM-1401
> URL: https://issues.apache.org/jira/browse/BEAM-1401
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1401) Sinks in Beam should supported windowed unbounded PCollections

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1401:
-

Assignee: Kenneth Knowles  (was: Davor Bonaci)

> Sinks in Beam should supported windowed unbounded PCollections
> --
>
> Key: BEAM-1401
> URL: https://issues.apache.org/jira/browse/BEAM-1401
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-892) revamp quickstart

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-892.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> revamp quickstart
> -
>
> Key: BEAM-892
> URL: https://issues.apache.org/jira/browse/BEAM-892
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Frances Perry
> Fix For: Not applicable
>
>
> We need to make this quickstart actually a quickstart!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-166) Version annotations in capability matrix

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-166:
--
Issue Type: Wish  (was: Bug)

> Version annotations in capability matrix
> 
>
> Key: BEAM-166
> URL: https://issues.apache.org/jira/browse/BEAM-166
> Project: Beam
>  Issue Type: Wish
>  Components: project-management, website
>Reporter: Tyler Akidau
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Particularly once we get to the point of having a real, usable release, the 
> capability matrix will be far more useful if we have annotations of which 
> versions the capabilities refer to, both Beam versions and dependency 
> versions (e.g. "requires Spark 2.0"). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-900) Spark quickstart instructions

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-900.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Spark quickstart instructions
> -
>
> Key: BEAM-900
> URL: https://issues.apache.org/jira/browse/BEAM-900
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Frances Perry
>Assignee: Amit Sela
> Fix For: Not applicable
>
>
> After initial quickstart structure is pushed, add commandlines for Spark 
> execution to quickstart.md and detailed Spark setup instructions to 
> learn/runners/spark.md



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1175) Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903550#comment-15903550
 ] 

Davor Bonaci commented on BEAM-1175:


I believe we have recently removed runner-facing Javadoc from being published 
for each release. Now, the published Javadoc should include only user facing 
things. (To became visible with the next release). I think this addresses the 
first part of this issue.

The second part is about making runner-facing Javadoc published. That is 
available on a per-module basis in Maven Central, and automatically integrated 
into some tooling. Not sure we need to publish it on the website just yet.

So, 1/2 is done, the other half can be deferred, I think. Resolving. Perhaps 
reopen if you think we need to address this in the near future.

> Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts
> ---
>
> Key: BEAM-1175
> URL: https://issues.apache.org/jira/browse/BEAM-1175
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: James Malone
> Fix For: Not applicable
>
>
> Currently, the runners-core Javadoc is included with the SDK's Javadoc on the 
> website.
> Pipeline authors should only be viewing the SDK's Javadoc, and likely the IOs.
> Generally, while I do think we can't really separate every IO Javadoc, it is 
> extremely confusing to have Javadoc for separate POM dependencies included 
> together, since the classes listed will not actually be available to someone 
> who just depends on the SDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1175) Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1175:
--

Assignee: Kenneth Knowles  (was: James Malone)

> Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts
> ---
>
> Key: BEAM-1175
> URL: https://issues.apache.org/jira/browse/BEAM-1175
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: Not applicable
>
>
> Currently, the runners-core Javadoc is included with the SDK's Javadoc on the 
> website.
> Pipeline authors should only be viewing the SDK's Javadoc, and likely the IOs.
> Generally, while I do think we can't really separate every IO Javadoc, it is 
> extremely confusing to have Javadoc for separate POM dependencies included 
> together, since the classes listed will not actually be available to someone 
> who just depends on the SDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1175) Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1175.

   Resolution: Fixed
Fix Version/s: Not applicable

> Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts
> ---
>
> Key: BEAM-1175
> URL: https://issues.apache.org/jira/browse/BEAM-1175
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: James Malone
> Fix For: Not applicable
>
>
> Currently, the runners-core Javadoc is included with the SDK's Javadoc on the 
> website.
> Pipeline authors should only be viewing the SDK's Javadoc, and likely the IOs.
> Generally, while I do think we can't really separate every IO Javadoc, it is 
> extremely confusing to have Javadoc for separate POM dependencies included 
> together, since the classes listed will not actually be available to someone 
> who just depends on the SDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1552) Beam contrib guide does not fully explain normal dev flow

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1552.

   Resolution: Fixed
Fix Version/s: Not applicable

> Beam contrib guide does not fully explain normal dev flow
> -
>
> Key: BEAM-1552
> URL: https://issues.apache.org/jira/browse/BEAM-1552
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: Not applicable
>
>
> I encountered some issues while starting to work on Beam and it'd be nice to 
> update the contributor's guide.
> The contrib guide contains pretty detailed git commands, but:
> 1. For pushing to your branch, the git commands don't actually work when you 
> try to use them in real life (you need to force push)
> 2. For squashing, we tell contributors that they can squash, but we don't 
> actually say that we expect them to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1367) Need a way to force a page to be set to a specific language for code blocks

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1367:
--

Assignee: (was: Davor Bonaci)

> Need a way to force a page to be set to a specific language for code blocks
> ---
>
> Key: BEAM-1367
> URL: https://issues.apache.org/jira/browse/BEAM-1367
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>
> When you have a language specific page that only has code blocks consisting 
> of that language and you are viewing it after toggling to a different 
> language, there is no indication there are any code blocks on the page at 
> all. Need a way to force a page to always be a specific language so the code 
> blocks always show up.
> Repro:
> 1. Go to programming guide and adapt for Java
> 2. Go to a page that only has python code blocks, and there is no indication 
> there are any code blocks at all. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1552) Beam contrib guide does not fully explain normal dev flow

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903538#comment-15903538
 ] 

Davor Bonaci commented on BEAM-1552:


PR got merged, so I assume this is done. Resolving. Feel free to reopen if 
additional action is necessary.

> Beam contrib guide does not fully explain normal dev flow
> -
>
> Key: BEAM-1552
> URL: https://issues.apache.org/jira/browse/BEAM-1552
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> I encountered some issues while starting to work on Beam and it'd be nice to 
> update the contributor's guide.
> The contrib guide contains pretty detailed git commands, but:
> 1. For pushing to your branch, the git commands don't actually work when you 
> try to use them in real life (you need to force push)
> 2. For squashing, we tell contributors that they can squash, but we don't 
> actually say that we expect them to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-977) Fill in Python SDK page

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903521#comment-15903521
 ] 

Davor Bonaci commented on BEAM-977:
---

Seems like already done: https://beam.apache.org/documentation/sdks/python/. 
Resolving.

> Fill in Python SDK page
> ---
>
> Key: BEAM-977
> URL: https://issues.apache.org/jira/browse/BEAM-977
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-792) In WC Walkthrough: Document how logging works on Apache Spark

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903534#comment-15903534
 ] 

Davor Bonaci commented on BEAM-792:
---

[~hadarhg], any more thoughts perhaps? Generally speaking, the walkthroughs 
should be runner-independent as much as possible.

> In WC Walkthrough: Document how logging works on Apache Spark
> -
>
> Key: BEAM-792
> URL: https://issues.apache.org/jira/browse/BEAM-792
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1650) Add IntelliJ code style xml to the project repo and website

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903530#comment-15903530
 ] 

Davor Bonaci commented on BEAM-1650:


[~aviemzur], what are you thoughts on how hard would it be to maintain this to 
be in sync with checkystyle?

> Add IntelliJ code style xml to the project repo and website 
> 
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa could benefit from having a code 
> style xml file which matches the checkstyle enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IntelliJ setup section: 
> https://beam.apache.org/contribute/contribution-guide/#intellij



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1650) Add IntelliJ code style xml to the project repo and website

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1650:
---
Component/s: (was: sdk-ideas)

> Add IntelliJ code style xml to the project repo and website 
> 
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa could benefit from having a code 
> style xml file which matches the checkstyle enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IntelliJ setup section: 
> https://beam.apache.org/contribute/contribution-guide/#intellij



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-791) In WC Walkthrough: Document how logging works on Apache Flink

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903524#comment-15903524
 ] 

Davor Bonaci commented on BEAM-791:
---

[~hadarhg], can you please provide any more details what are you looking for 
here?

> In WC Walkthrough: Document how logging works on Apache Flink
> -
>
> Key: BEAM-791
> URL: https://issues.apache.org/jira/browse/BEAM-791
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-834) Add HTML/md versions of technical reference docs

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-834:
-

Assignee: (was: Kenneth Knowles)

> Add HTML/md versions of technical reference docs
> 
>
> Key: BEAM-834
> URL: https://issues.apache.org/jira/browse/BEAM-834
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Kenneth Knowles
>
> We have a number of technical docs that might be useful as a deep dive - 
> mostly aimed at contributors but potentially important for power users. These 
> are not the 1-pagers and design docs, but the descriptions of existing 
> features or the project. Candidates include:
>  - Some parts of [Triggers in Apache Beam 
> (incubating)|https://s.apache.org/beam-triggers]
>  - Probably much of [Lateness in Apache Beam 
> (incubating)|https://s.apache.org/beam-lateness]
>  - Some bits of [Static Display 
> Data|https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing]
>  - Some pieces of the [Technical 
> Vision|https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPgXNQA1oD-rGooWOSwAqh8/edit?usp=sharing]?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-667) Include code snippets from real examples

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-667:
--
Fix Version/s: First stable release

> Include code snippets from real examples
> 
>
> Key: BEAM-667
> URL: https://issues.apache.org/jira/browse/BEAM-667
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Frances Perry
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-977) Fill in Python SDK page

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-977.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Fill in Python SDK page
> ---
>
> Key: BEAM-977
> URL: https://issues.apache.org/jira/browse/BEAM-977
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1013:
---
Fix Version/s: First stable release

> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1015) Need language specific tabs that allow both regular text and code blocks

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1015:
--

Assignee: (was: James Malone)

> Need language specific tabs that allow both regular text and code blocks
> 
>
> Key: BEAM-1015
> URL: https://issues.apache.org/jira/browse/BEAM-1015
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Melissa Pashniak
>
> Right now, the Java/python/etc. toggle tabs work well for code blocks, but if 
> you need to have some accompanying language specific text with your code 
> block it becomes unwieldy, as everything in there is treated as code. Ideally 
> we could have both regular text and code blocks inside the language toggle 
> tabs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-916) Update the "Using the Documentation" doc (/documentation/)

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-916.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Update the "Using the Documentation" doc (/documentation/)
> --
>
> Key: BEAM-916
> URL: https://issues.apache.org/jira/browse/BEAM-916
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-504) Fill in the documentation/sdks/java portion of the website

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-504.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Fill in the documentation/sdks/java portion of the website
> --
>
> Key: BEAM-504
> URL: https://issues.apache.org/jira/browse/BEAM-504
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Java-SDK-specific content like existing IO 
> connectors, javadoc, etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-504) Fill in the documentation/sdks/java portion of the website

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-504:
-

Assignee: Hadar Hod  (was: James Malone)

> Fill in the documentation/sdks/java portion of the website
> --
>
> Key: BEAM-504
> URL: https://issues.apache.org/jira/browse/BEAM-504
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Java-SDK-specific content like existing IO 
> connectors, javadoc, etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-916) Update the "Using the Documentation" doc (/documentation/)

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903512#comment-15903512
 ] 

Davor Bonaci commented on BEAM-916:
---

Seems like already done: https://beam.apache.org/documentation/. Resolving.

> Update the "Using the Documentation" doc (/documentation/)
> --
>
> Key: BEAM-916
> URL: https://issues.apache.org/jira/browse/BEAM-916
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #11

[Apache Jenkins Server]

See 


--
[...truncated 473.55 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.245 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.095 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.189 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running 

[jira] [Commented] (BEAM-509) Fill in the Additonal Resources page of the website

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903507#comment-15903507
 ] 

Davor Bonaci commented on BEAM-509:
---

Seems like it is done: https://beam.apache.org/documentation/resources/. 
Resolving.

> Fill in the Additonal Resources page of the website
> ---
>
> Key: BEAM-509
> URL: https://issues.apache.org/jira/browse/BEAM-509
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Frances Perry
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit
> Do a nicer curation of great Beam articles, videos, etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #11

[Apache Jenkins Server]

See 


--
[...truncated 474.55 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.288 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.23 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.172 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running 

[jira] [Commented] (BEAM-1660) withCoder() error in JdbcIO JavaDoc example

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903486#comment-15903486
 ] 

Xu Mingmin commented on BEAM-1660:
--

create a new task BEAM-1675 to deprecate withCoder() in JdbcIO

> withCoder() error in JdbcIO JavaDoc example
> ---
>
> Key: BEAM-1660
> URL: https://issues.apache.org/jira/browse/BEAM-1660
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: JavaDoc, JdbcIO
>
> Follow the JavaDoc of JdbcIO, job fails with exception:
> {code}
> Exception in thread "main" java.lang.IllegalStateException: JdbcIO.read() 
> requires a coder to be set via withCoder(coder)
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:176)
>   at org.apache.beam.sdk.io.jdbc.JdbcIO$Read.validate(JdbcIO.java:329)
>   at org.apache.beam.sdk.io.jdbc.JdbcIO$Read.validate(JdbcIO.java:249)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:419)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:350)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:172)
> {code}
> It requires a Coder provided by withCoder. Need to add it in the example.
> Other point is, does it really need to specify a Coder? Users should register 
> it with CoderRegistry I think.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #174: Finding a typo in a code block

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/174


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: Finding a typo in a code block

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 6dd302c0f -> 7b78cc342


Finding a typo in a code block

I found a typo in the Markdown code block. The ``` should be gone.


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/9e1f59f9
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/9e1f59f9
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/9e1f59f9

Branch: refs/heads/asf-site
Commit: 9e1f59f995fe8eaef705c1bd8e5ce2897fb17699
Parents: 6dd302c
Author: 성준영 
Authored: Thu Mar 9 17:53:27 2017 +0900
Committer: Ahmet Altay 
Committed: Thu Mar 9 09:53:25 2017 -0800

--
 src/documentation/programming-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/9e1f59f9/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 0aca04e..81a03fb 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -1090,7 +1090,7 @@ PCollection numbers =
   p.begin()
   .apply(TextIO.Read.named("ReadNumbers")
 .from("gs://my_bucket/path/to/numbers-*.txt")
-.withCoder(TextualIntegerCoder.of()));```
+.withCoder(TextualIntegerCoder.of()));
 ```
 
 ```py

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/51425e85
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/51425e85
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/51425e85

Branch: refs/heads/asf-site
Commit: 51425e85ee9644a1547557364f0258b78669886f
Parents: 9e1f59f
Author: Ahmet Altay 
Authored: Thu Mar 9 09:54:43 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Mar 9 09:54:43 2017 -0800

--
 content/documentation/programming-guide/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/51425e85/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index 0752642..50bf398 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -1390,7 +1390,7 @@ tree, [2]
   p.begin()
   .apply(TextIO.Read.named("ReadNumbers")
 .from("gs://my_bucket/path/to/numbers-*.txt")
-.withCoder(TextualIntegerCoder.of()));```
+.withCoder(TextualIntegerCoder.of()));
 
 
 

[jira] [Created] (BEAM-1675) deprecate withCoder() in JdbcIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1675:


 Summary: deprecate withCoder() in JdbcIO
 Key: BEAM-1675
 URL: https://issues.apache.org/jira/browse/BEAM-1675
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Davor Bonaci


In JdbcIO.read(), a Coder is required to handle the serialize/deserialize of 
input record similar as below example. It's better to leverage Coders defined 
in CoderRegistry.  

{code}
.withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-357) Build doesn't pass on windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903470#comment-15903470
 ] 

ASF GitHub Bot commented on BEAM-357:
-

Github user rmannibucau closed the pull request at:

https://github.com/apache/beam/pull/496


> Build doesn't pass on windows
> -
>
> Key: BEAM-357
> URL: https://issues.apache.org/jira/browse/BEAM-357
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, runner-spark, sdk-java-extensions
>Reporter: Romain Manni-Bucau
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.2.0-incubating
>
> Attachments: bean-win.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1640) data file missing when submit a job on Flink

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903472#comment-15903472
 ] 

Xu Mingmin commented on BEAM-1640:
--

[~aljoscha], it fails with Flink 1.2.0, Beam 0.6.0-SNAPSHOT. The same code 
works with Flink 1.1.4, which is Beam 0.5.0

> data file missing when submit a job on Flink
> 
>
> Key: BEAM-1640
> URL: https://issues.apache.org/jira/browse/BEAM-1640
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.6.0
>Reporter: Xu Mingmin
>Assignee: Aljoscha Krettek
>
> I've one file with path 'META-INF/jaas/kafka_jaas.conf' in my jar package. it 
> works with Beam 0.5.0, when I re-package it with 0.6.0-SNAPSHOT, it fails to 
> submit with bin/flink command. --Both run on YARN.
> The error is show as below, I guess this file maybe lost in Flink-Runner. 
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> consumer
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:702)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:557)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:540)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$2.apply(KafkaIO.java:503)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$2.apply(KafkaIO.java:501)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaSource.generateInitialSplits(KafkaIO.java:620)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.(UnboundedSourceWrapper.java:159)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:267)
>   ... 33 more
> Caused by: org.apache.kafka.common.KafkaException: 
> java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in 
> jaas config.
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
>   at 
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:623)
>   ... 40 more
> Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' 
> entry in jaas config.
>   at io.ebay.rheos.kafka.security.iaf.IAFLogin.login(IAFLogin.java:54)
>   at 
> org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:53)
>   at 
> org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:75)
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
>   ... 43 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #496: [BEAM-357] fixing build on windows

[rmannibucau]

Github user rmannibucau closed the pull request at:

https://github.com/apache/beam/pull/496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1871

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1562) Use a "signal" to stop streaming tests as they finish.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1562.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Use a "signal" to stop streaming tests as they finish.
> --
>
> Key: BEAM-1562
> URL: https://issues.apache.org/jira/browse/BEAM-1562
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: First stable release
>
>
> Streaming tests use a timeout that has to take a large enough buffer to avoid 
> slow runtimes stopping before test completes.
> We can introduce a "poison pill" based on an {{EndOfStream}} element that 
> would be counted in a Metrics/Aggregator to know all data was processed.
> Another option would be to follow the slowest WM and stop once it hits 
> end-of-time.
> This requires the Spark runner to stop blocking the execution of a streaming 
> pipeline until it's complete - which is something relevant to 
> {{PipelineResult}} (re)design that is being discussed in BEAM-849   



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1582.
-
   Resolution: Fixed
Fix Version/s: First stable release

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1184) Add integration tests for ElasticsearchIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-1184:
---
Fix Version/s: First stable release

> Add integration tests for ElasticsearchIO
> -
>
> Key: BEAM-1184
> URL: https://issues.apache.org/jira/browse/BEAM-1184
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Etienne Chauchot
> Fix For: First stable release
>
>
> In https://github.com/apache/incubator-beam/pull/1439#pullrequestreview, the 
> original PR included a dockerfile and an integration test that ran against an 
> ES instance in that dockerfile. 
> We are working on infrastructure to stand up docker containers for this kind 
> of testing, but aren't quite ready for it, so we pulled the scripts, 
> dockerfile+IT out of that PR. This issue tracks checking that work in once we 
> can take advantage of it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #10

[Apache Jenkins Server]

See 


Changes:

[amitsela33] Test runner to stop on EOT watermark, or timeout.

--
[...truncated 475.15 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.225 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.085 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #10

[Apache Jenkins Server]

See 


Changes:

[amitsela33] Test runner to stop on EOT watermark, or timeout.

--
[...truncated 473.62 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.337 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.059 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.157 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.122 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #10

[Apache Jenkins Server]

See 


Changes:

[amitsela33] Test runner to stop on EOT watermark, or timeout.

--
[...truncated 474.60 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.055 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, 

[2/2] beam git commit: This closes #2168

[amitsela]

This closes #2168


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/efc701ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/efc701ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/efc701ed

Branch: refs/heads/master
Commit: efc701ed614f137129bb33544ad83d61ee9d0278
Parents: 7954896 2e30846
Author: Amit Sela 
Authored: Thu Mar 9 18:03:40 2017 +0200
Committer: Amit Sela 
Committed: Thu Mar 9 18:03:40 2017 +0200

--
 .../beam/runners/spark/SparkPipelineResult.java | 52 --
 .../runners/spark/TestSparkPipelineOptions.java | 26 +
 .../beam/runners/spark/TestSparkRunner.java | 53 +--
 .../spark/stateful/SparkTimerInternals.java | 10 +++-
 .../apache/beam/runners/spark/PipelineRule.java | 33 
 .../runners/spark/SparkPipelineStateTest.java   | 12 ++---
 .../translation/streaming/CreateStreamTest.java | 11 ++--
 .../ResumeFromCheckpointStreamingTest.java  | 55 ++--
 8 files changed, 172 insertions(+), 80 deletions(-)
--

[GitHub] beam pull request #2168: [BEAM-1582, BEAM-1562] Stop streaming tests on EOT ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2168


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903277#comment-15903277
 ] 

ASF GitHub Bot commented on BEAM-1582:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2168


> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Test runner to stop on EOT watermark, or timeout.

[amitsela]

Repository: beam
Updated Branches:
  refs/heads/master 7954896a5 -> efc701ed6


Test runner to stop on EOT watermark, or timeout.

Remove timeout since it is already a pipeline option.

Advance to infinity at the end of pipelines.

Add EOT watermark and expected assertions test options.

SparkPipelineResult should avoid returning null, and handle exceptions better.

Make ResumeFromCheckpointStreamingTest use TestSparkRunner and stop on EOT 
watermark.

Stop the context and update the state in finally.

Addressed comments - better name for a watermark that stops execution.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e308463
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e308463
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e308463

Branch: refs/heads/master
Commit: 2e3084631dbb56acd3971f12596953b63290c9f2
Parents: 7954896
Author: Sela 
Authored: Sat Mar 4 21:04:02 2017 +0200
Committer: Amit Sela 
Committed: Thu Mar 9 18:02:39 2017 +0200

--
 .../beam/runners/spark/SparkPipelineResult.java | 52 --
 .../runners/spark/TestSparkPipelineOptions.java | 26 +
 .../beam/runners/spark/TestSparkRunner.java | 53 +--
 .../spark/stateful/SparkTimerInternals.java | 10 +++-
 .../apache/beam/runners/spark/PipelineRule.java | 33 
 .../runners/spark/SparkPipelineStateTest.java   | 12 ++---
 .../translation/streaming/CreateStreamTest.java | 11 ++--
 .../ResumeFromCheckpointStreamingTest.java  | 55 ++--
 8 files changed, 172 insertions(+), 80 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2e308463/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
index ab59fb2..ddc1964 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
@@ -19,6 +19,7 @@
 package org.apache.beam.runners.spark;
 
 import java.io.IOException;
+import java.util.Objects;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
@@ -56,11 +57,11 @@ public abstract class SparkPipelineResult implements 
PipelineResult {
 state = State.RUNNING;
   }
 
-  private RuntimeException runtimeExceptionFrom(final Throwable e) {
+  private static RuntimeException runtimeExceptionFrom(final Throwable e) {
 return (e instanceof RuntimeException) ? (RuntimeException) e : new 
RuntimeException(e);
   }
 
-  private RuntimeException beamExceptionFrom(final Throwable e) {
+  private static RuntimeException beamExceptionFrom(final Throwable e) {
 // Scala doesn't declare checked exceptions in the bytecode, and the Java 
compiler
 // won't let you catch something that is not declared, so we can't catch
 // SparkException directly, instead we do an instanceof check.
@@ -107,15 +108,15 @@ public abstract class SparkPipelineResult implements 
PipelineResult {
 try {
   state = awaitTermination(duration);
 } catch (final TimeoutException e) {
-  state = null;
+  // ignore.
 } catch (final ExecutionException e) {
   state = PipelineResult.State.FAILED;
+  stop();
   throw beamExceptionFrom(e.getCause());
 } catch (final Exception e) {
   state = PipelineResult.State.FAILED;
-  throw beamExceptionFrom(e);
-} finally {
   stop();
+  throw beamExceptionFrom(e);
 }
 
 return state;
@@ -149,6 +150,9 @@ public abstract class SparkPipelineResult implements 
PipelineResult {
 @Override
 protected void stop() {
   SparkContextFactory.stopSparkContext(javaSparkContext);
+  if (Objects.equals(state, State.RUNNING)) {
+state = State.STOPPED;
+  }
 }
 
 @Override
@@ -178,19 +182,37 @@ public abstract class SparkPipelineResult implements 
PipelineResult {
   // after calling stop, if exception occurs in "grace period" it won't 
propagate.
   // calling the StreamingContext's waiter with 0 msec will throw any 
error that might have
   // been thrown during the "grace period".
-  javaStreamingContext.awaitTermination(0);
-  SparkContextFactory.stopSparkContext(javaSparkContext);
+  try {
+javaStreamingContext.awaitTermination(0);
+  } catch (Exception e) {
+throw beamExceptionFrom(e);
+  } finally {
+SparkContextFactory.stopSparkContext(javaSparkContext);
+if 

[jira] [Commented] (BEAM-1674) Flink user state GC depends on order of timer firing

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903196#comment-15903196
 ] 

Aljoscha Krettek commented on BEAM-1674:


[~lzljs3620320] Do you maybe have an idea about this?

I think we can add a complicated scheme of piggybacking several timers on one 
"physical" Flink timer but we would need to manage those and keep some state to 
determine what timer ids we have to fire for when a physical Flink timer fires.

> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1674) Flink user state GC depends on order of timer firing

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-1674:
---
Summary: Flink user state GC depends on order of timer firing  (was: User 
state GC depends on order of timer firing)

> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1674) User state GC depends on order of timer firing

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903183#comment-15903183
 ] 

Aljoscha Krettek commented on BEAM-1674:


[~kenn] I'm afraid we can't simply use timers with different ids to separate 
cleanup timers from the user timers. This will affect everyone using the 
{{StatefulDoFnRunner}} and you mentioned that you might also want to move the 
Dataflow Runner over to it. 

> User state GC depends on order of timer firing
> --
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1674) User state GC depends on order of timer firing

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-1674:
--

 Summary: User state GC depends on order of timer firing
 Key: BEAM-1674
 URL: https://issues.apache.org/jira/browse/BEAM-1674
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.5.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Priority: Blocker
 Fix For: 0.6.0


The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
be the GC timer for the user state. The Flink Runner uses timers to schedule 
GC, the user-set timer and the GC timer have a different timer id, so they 
don't clash. However, if the GC timer is being processed before the user timer 
then the user doesn't have a chance to access the state anymore because it will 
already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-68) Support for limiting parallelism of a step

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903135#comment-15903135
 ] 

Eugene Kirpichov commented on BEAM-68:
--

I think this needs a discussion on the Beam dev@ mailing list. We should have a 
general approach to annotating pipeline elements with runner-specific 
information. I don't think "annotate the pipeline and let runners ignore it" is 
a good approach; the main reason being that this would violate the abstraction 
boundary, where a pipeline is first constructed in a runner-agnostic way (in 
fact while the runner is not even available), and then run. E.g. the set of all 
possible runner-specific annotations is not known in advance: while "step 
parallelism limit" seems relatively generic, suppose if say Apex allowed you to 
set an "Apex frobnication level" parameter on a step - it would look pretty 
weird if the Beam pipeline API had a withApexFrobnicationLevel method on a 
ParDo, and would introduce an illegal dependency from Beam to Apex.

> Support for limiting parallelism of a step
> --
>
> Key: BEAM-68
> URL: https://issues.apache.org/jira/browse/BEAM-68
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets 
> TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected 
> QPS/step) on the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. 
> A GroupByKey with exactly k keys will guarantee that only k elements are 
> produced, but runners are free to break fusion in ways that each element may 
> be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the 
> Beam Model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1660) withCoder() error in JdbcIO JavaDoc example

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903124#comment-15903124
 ] 

Eugene Kirpichov commented on BEAM-1660:


Agreed about the coder issue: we should take advantage of coder inference if 
possible. Though that should probably be a separate JIRA, and a feature request 
rather than a bug.

> withCoder() error in JdbcIO JavaDoc example
> ---
>
> Key: BEAM-1660
> URL: https://issues.apache.org/jira/browse/BEAM-1660
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: JavaDoc, JdbcIO
>
> Follow the JavaDoc of JdbcIO, job fails with exception:
> {code}
> Exception in thread "main" java.lang.IllegalStateException: JdbcIO.read() 
> requires a coder to be set via withCoder(coder)
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:176)
>   at org.apache.beam.sdk.io.jdbc.JdbcIO$Read.validate(JdbcIO.java:329)
>   at org.apache.beam.sdk.io.jdbc.JdbcIO$Read.validate(JdbcIO.java:249)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:419)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:350)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:172)
> {code}
> It requires a Coder provided by withCoder. Need to add it in the example.
> Other point is, does it really need to specify a Coder? Users should register 
> it with CoderRegistry I think.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2506

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1640) data file missing when submit a job on Flink

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903038#comment-15903038
 ] 

Aljoscha Krettek commented on BEAM-1640:


Which Flink version are you using?

> data file missing when submit a job on Flink
> 
>
> Key: BEAM-1640
> URL: https://issues.apache.org/jira/browse/BEAM-1640
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.6.0
>Reporter: Xu Mingmin
>Assignee: Aljoscha Krettek
>
> I've one file with path 'META-INF/jaas/kafka_jaas.conf' in my jar package. it 
> works with Beam 0.5.0, when I re-package it with 0.6.0-SNAPSHOT, it fails to 
> submit with bin/flink command. --Both run on YARN.
> The error is show as below, I guess this file maybe lost in Flink-Runner. 
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> consumer
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:702)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:557)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:540)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$2.apply(KafkaIO.java:503)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$2.apply(KafkaIO.java:501)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaSource.generateInitialSplits(KafkaIO.java:620)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.(UnboundedSourceWrapper.java:159)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:267)
>   ... 33 more
> Caused by: org.apache.kafka.common.KafkaException: 
> java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in 
> jaas config.
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
>   at 
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:623)
>   ... 40 more
> Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' 
> entry in jaas config.
>   at io.ebay.rheos.kafka.security.iaf.IAFLogin.login(IAFLogin.java:54)
>   at 
> org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:53)
>   at 
> org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:75)
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
>   ... 43 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2854

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-1673) PubSubIO can't write attributes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1673:
--

Assignee: Jean-Baptiste Onofré  (was: Daniel Halperin)

> PubSubIO can't write attributes
> ---
>
> Key: BEAM-1673
> URL: https://issues.apache.org/jira/browse/BEAM-1673
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.5.0
>Reporter: Bin Chen
>Assignee: Jean-Baptiste Onofré
>
> PubSubIO ignore the attributes of message when writing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1870

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #9

[Apache Jenkins Server]

See 


--
[...truncated 475.00 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.279 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.075 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.138 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.172 sec - in 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #9

[Apache Jenkins Server]

See 


--
[...truncated 474.59 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.09 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.988 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.092 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.172 sec - in 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1461

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Flink #1869

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2210: [BEAM-1674] Add unicode type to the typeDict attrib...

[yuya-tajima]

GitHub user yuya-tajima opened a pull request:

https://github.com/apache/beam/pull/2210

[BEAM-1674] Add unicode type to the typeDict attribute in Python SDK

When I executed Biq Query exection on the Google Cloud Dataflow,  the 
following error occurred.

```
UnicodeEncodeError: 'ascii' codec can't encode characters in position 
73-78: ordinal not in range(128)
```

It seems there is a problem when the executed SQL is UTF-8 strings(unicode 
type).

```python:apache_beam/transforms/display.py
#TODO: Fix Args: documentation once the Python classes handling has 
changed
type_ = cls.typeDict.get(type(value))
if type_ is None:
  type_ = 'CLASS' if inspect.isclass(value) else None
if type_ is None and value is None:
  type_ = 'STRING'
return type_
```

so unicode type is always considered as None type.

So I fixed the problem the following code.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yuya-tajima/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2210.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2210


commit 948b802fd35b1f810de30829fc2e46fcdbe6ce79
Author: tajima 
Date:   2017-03-09T09:08:25Z

add unicode type to the typeDict attribute in Python SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1641) Support synchronized processing time in Flink runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902802#comment-15902802
 ] 

Aljoscha Krettek commented on BEAM-1641:


Wow, that's a tough one. I think we might first have to introduce such a 
concept in Flink if we want to use it from Beam.

> Support synchronized processing time in Flink runner
> 
>
> Key: BEAM-1641
> URL: https://issues.apache.org/jira/browse/BEAM-1641
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Aljoscha Krettek
>
> The "continuation trigger" for a processing time trigger is a synchronized 
> processing time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and 
> that element arrives at GBK2, it will wait until all the other upstream keys 
> have also processed and emitted corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1673) PubSubIO can't write attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902747#comment-15902747
 ] 

ASF GitHub Bot commented on BEAM-1673:
--

GitHub user bchen-talend opened a pull request:

https://github.com/apache/beam/pull/2209

[BEAM-1673] Fix PubSubIO write attribute issue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bchen-talend/incubator-beam 
bugfix/PubSubIO-write-attributes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2209


commit 87d02d72051e3afd15e38b31445704572d15034a
Author: Chen Bin 
Date:   2017-03-09T03:09:04Z

Fix PubSubIO write attribute issue




> PubSubIO can't write attributes
> ---
>
> Key: BEAM-1673
> URL: https://issues.apache.org/jira/browse/BEAM-1673
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.5.0
>Reporter: Bin Chen
>Assignee: Daniel Halperin
>
> PubSubIO ignore the attributes of message when writing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2209: [BEAM-1673] Fix PubSubIO write attribute issue

[bchen-talend]

GitHub user bchen-talend opened a pull request:

https://github.com/apache/beam/pull/2209

[BEAM-1673] Fix PubSubIO write attribute issue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bchen-talend/incubator-beam 
bugfix/PubSubIO-write-attributes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2209


commit 87d02d72051e3afd15e38b31445704572d15034a
Author: Chen Bin 
Date:   2017-03-09T03:09:04Z

Fix PubSubIO write attribute issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1673) PubSubIO can't write attributes

[Bin Chen (JIRA)]

Bin Chen created BEAM-1673:
--

 Summary: PubSubIO can't write attributes
 Key: BEAM-1673
 URL: https://issues.apache.org/jira/browse/BEAM-1673
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 0.5.0
Reporter: Bin Chen
Assignee: Daniel Halperin


PubSubIO ignore the attributes of message when writing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #174: Finding a typo in a code block

[sungjunyoung]

GitHub user sungjunyoung opened a pull request:

https://github.com/apache/beam-site/pull/174

Finding a typo in a code block

I found a typo in the Markdown code block. The ``` should be gone.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sungjunyoung/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #174


commit c5b7b54efcd2711c001331ad3512069dfb537d2e
Author: 성준영 
Date:   2017-03-09T08:53:27Z

Finding a typo in a code block

I found a typo in the Markdown code block. The ``` should be gone.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---