[
https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248925#comment-15248925
]
ASF GitHub Bot commented on BEAM-53:
------------------------------------
GitHub user mshields822 opened a pull request:
https://github.com/apache/incubator-beam/pull/213
[BEAM-53] Pubsub Apiary client
Continuing from https://github.com/apache/incubator-beam/pull/120
- Move PubsubClient and friends to sdk.util.
- Add PubsubApiaryClient since gRPC has onerous boot class path
requirements which I did not wish to inflict upon other runners.
- Unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mshields822/incubator-beam pubsub-apiary
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/213.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #213
----
commit 4e2a1014524e3dd034fc88334949db16653e8010
Author: Mark Shields <[email protected]>
Date: 2016-04-13T23:44:30Z
PubsubClient improvements
- Move to util/
- PubsubClient becomes abstract class.
- More name checking (working towards removing all magic from PubsubIO)
- Add apiary client since gRPC is such a pain to support
- Make create generic.
commit 34c93cc1f9fe70251564289fcc60831ab5736d67
Author: Mark Shields <[email protected]>
Date: 2016-04-16T03:52:17Z
Batch PubsubIO uses PubsubApiaryClient. Port unit tests
commit 6ebfa8b28b650f07eab4e8762721e80af281bdbe
Author: Mark Shields <[email protected]>
Date: 2016-04-19T22:41:13Z
Unit tests.
commit 26938b56698ea7f8e0996c34b921ba666d4a059b
Author: Mark Shields <[email protected]>
Date: 2016-04-19T23:10:36Z
Import fiddles.
----
> PubSubIO: reimplement in Java
> -----------------------------
>
> Key: BEAM-53
> URL: https://issues.apache.org/jira/browse/BEAM-53
> Project: Beam
> Issue Type: New Feature
> Components: runner-core
> Reporter: Daniel Halperin
> Assignee: Mark Shields
>
> PubSubIO is currently only partially implemented in Java: the
> DirectPipelineRunner uses a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path
> implemented in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in
> a scalable way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than
> from a message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
