[GitHub] incubator-beam pull request: [BEAM-82] Exclude old version of Goog...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/7

[BEAM-82] Exclude old version of Google API client libraries

Our com.google.apis:google-api-services-datastore-protobuf dependency
has a transitive dependency on a very old Google API client library.
This change suppresses that dependency and replaces it by depending
on a version that matches our other API dependencies.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit d92e83a589724cc7d972f1f9e2b9395cb67dcf55
Author: Kenneth Knowles 
Date:   2016-03-01T07:11:42Z

Exclude old version of Google API client libraries

Our com.google.apis:google-api-services-datastore-protobuf dependency
has a transitive dependency on a very old Google API client library.
This change suppresses that dependency and replaces it by depending
on a version that matches our other API dependencies.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-83) Beam SDK has some difficulties upon import to Eclipse as maven project

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173334#comment-15173334
 ] 

ASF GitHub Bot commented on BEAM-83:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/6

[BEAM-83] Move Java 8 tests to their own module

This allows easy setting of the compiler source and target version to 1.8 
without any fragile plugin configuration. It also allows the import into 
Eclipse to easily set separate compliance levels, as this
is done per-Eclipse-project.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam Java8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/6.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6


commit af15101c9632375bc2ef09b81bde084aa1d22b8c
Author: Kenneth Knowles 
Date:   2016-03-01T06:40:01Z

Move Java 8 tests to their own module

This allows easy setting of the compiler source and target version to
1.8 without any fragile plugin configuration. It also allows the
import into Eclipse to easily set separate compliance levels, as this
is done per-Eclipse-project.




> Beam SDK has some difficulties upon import to Eclipse as maven project
> --
>
> Key: BEAM-83
> URL: https://issues.apache.org/jira/browse/BEAM-83
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: Eclipse, Java8
>
> Upon first import of the project to Eclipse, the various Java 8 source 
> directories result in errors since compliance is project-wide for Eclipse. 
> The fix of upgrading compliance to version 1.8 results in some compile errors 
> in some areas of the code, particularly those that use unchecked casts, 
> rawtypes, and generic return type without any parameter to bind the actual 
> type parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-83) Beam SDK has some difficulties upon import to Eclipse as maven project

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-83:
---

 Summary: Beam SDK has some difficulties upon import to Eclipse as 
maven project
 Key: BEAM-83
 URL: https://issues.apache.org/jira/browse/BEAM-83
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Upon first import of the project to Eclipse, the various Java 8 source 
directories result in errors since compliance is project-wide for Eclipse. The 
fix of upgrading compliance to version 1.8 results in some compile errors in 
some areas of the code, particularly those that use unchecked casts, rawtypes, 
and generic return type without any parameter to bind the actual type parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173229#comment-15173229
 ] 

Frances Perry commented on BEAM-79:
---

Happy to assign it to you, as you will clearly be the expert on Gearpump ;-)

But please note that Beam is still very much under construction and there are a 
number of breaking API changes likely in the near future. So please reach out 
before getting beyond the early design phase / determining how well the models 
align. If you haven't yet, I'd start with these resources: 
http://beam.incubator.apache.org/getting_started/


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-79) Gearpump runner

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-79:
--
Assignee: Manu Zhang  (was: James Malone)

> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-79) Gearpump runner

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-79:
--
Assignee: James Malone

> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: James Malone
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-82) Transitive dependencies in Beam pom.xml have conflicts that could result in old versions if reordered

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-82:
---

 Summary: Transitive dependencies in Beam pom.xml have conflicts 
that could result in old versions if reordered
 Key: BEAM-82
 URL: https://issues.apache.org/jira/browse/BEAM-82
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Specifically, com.google.apis:google-api-services-datastore-protobuf depends on 
a very old version of the suite of Google API libraries. It is by maven's 
dependency resolution that these are generally overridden by other dependencies 
on the new versions.

It is easy (and I have done it) to get things rearranged so that it pulls in 
the very old API clients. They should be suppressed and provided in their 
latest form for compatibility with the rest of the SDK that is up to date.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-81) --update PipelineOption is in the wrong interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172919#comment-15172919
 ] 

ASF GitHub Bot commented on BEAM-81:


GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/5

[BEAM-81] --update PipelineOption is in the wrong interface



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam b23122757

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/5.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5


commit f7889e1f9c0270442812e50f6f4e638d9977cd99
Author: Scott Wegner 
Date:   2016-02-29T23:24:49Z

Move 'update' pipeline option to correct interface




> --update PipelineOption is in the wrong interface
> -
>
> Key: BEAM-81
> URL: https://issues.apache.org/jira/browse/BEAM-81
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> Right now, --update PipelineOption is part of the 
> DataflowPipelineDebugOptions. It doesn't have anything to do with "debugging".
> The entire DataflowPipelineDebugOptions is marked Internal and @Hidden.
>   @Hidden will make it not appear in '--help'.
>   Internal will make it marked as such in Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-81] --update PipelineOption is ...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/5

[BEAM-81] --update PipelineOption is in the wrong interface



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam b23122757

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/5.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5


commit f7889e1f9c0270442812e50f6f4e638d9977cd99
Author: Scott Wegner 
Date:   2016-02-29T23:24:49Z

Move 'update' pipeline option to correct interface




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-81) --update PipelineOption is in the wrong interface

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-81:
-
Assignee: Scott Wegner  (was: Davor Bonaci)

> --update PipelineOption is in the wrong interface
> -
>
> Key: BEAM-81
> URL: https://issues.apache.org/jira/browse/BEAM-81
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> Right now, --update PipelineOption is part of the 
> DataflowPipelineDebugOptions. It doesn't have anything to do with "debugging".
> The entire DataflowPipelineDebugOptions is marked Internal and @Hidden.
>   @Hidden will make it not appear in '--help'.
>   Internal will make it marked as such in Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-81) --update PipelineOption is in the wrong interface

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-81:


 Summary: --update PipelineOption is in the wrong interface
 Key: BEAM-81
 URL: https://issues.apache.org/jira/browse/BEAM-81
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Scott Wegner
Assignee: Davor Bonaci
Priority: Minor


Right now, --update PipelineOption is part of the DataflowPipelineDebugOptions. 
It doesn't have anything to do with "debugging".

The entire DataflowPipelineDebugOptions is marked Internal and @Hidden.
  @Hidden will make it not appear in '--help'.
  Internal will make it marked as such in Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-80) Support combiner lifting for (Keyed)CombineWithContext

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-80:
-
Assignee: Pei He  (was: Davor Bonaci)

> Support combiner lifting for (Keyed)CombineWithContext
> --
>
> Key: BEAM-80
> URL: https://issues.apache.org/jira/browse/BEAM-80
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>
> This is a missing feature of combine with context.
> Combiner lifting is currently disabled for (Keyed)CombineWithContext with a 
> passing through ParDo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-52) KafkaIO - bounded/unbounded, source/sink

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172824#comment-15172824
 ] 

Davor Bonaci commented on BEAM-52:
--

Assigned.

> KafkaIO - bounded/unbounded, source/sink
> 
>
> Key: BEAM-52
> URL: https://issues.apache.org/jira/browse/BEAM-52
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Daniel Halperin
>Assignee: Raghu Angadi
>
> We should support Apache Kafka. The priority list is probably:
> * UnboundedSource
> * unbounded Sink
> * BoundedSource
> * bounded Sink
> The connector should be well-tested, especially around UnboundedSource 
> checkpointing and resuming, and data duplication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-52) KafkaIO - bounded/unbounded, source/sink

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172735#comment-15172735
 ] 

Jean-Baptiste Onofré commented on BEAM-52:
--

It should also have options for partition sourcing and sinking.

> KafkaIO - bounded/unbounded, source/sink
> 
>
> Key: BEAM-52
> URL: https://issues.apache.org/jira/browse/BEAM-52
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Daniel Halperin
>
> We should support Apache Kafka. The priority list is probably:
> * UnboundedSource
> * unbounded Sink
> * BoundedSource
> * bounded Sink
> The connector should be well-tested, especially around UnboundedSource 
> checkpointing and resuming, and data duplication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-52) KafkaIO - bounded/unbounded, source/sink

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172622#comment-15172622
 ] 

Raghu Angadi commented on BEAM-52:
--

[~dhalp...@google.com], please assign the jira to me.

> KafkaIO - bounded/unbounded, source/sink
> 
>
> Key: BEAM-52
> URL: https://issues.apache.org/jira/browse/BEAM-52
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Daniel Halperin
>
> We should support Apache Kafka. The priority list is probably:
> * UnboundedSource
> * unbounded Sink
> * BoundedSource
> * bounded Sink
> The connector should be well-tested, especially around UnboundedSource 
> checkpointing and resuming, and data duplication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)