Alec Ferguson created BEAM-9491:
-----------------------------------
Summary: DatastoreV1.write() Should Optionally Return
PCollection<Void>
Key: BEAM-9491
URL: https://issues.apache.org/jira/browse/BEAM-9491
Project: Beam
Issue Type: Improvement
Components: io-java-gcp
Affects Versions: 2.20.0
Reporter: Alec Ferguson
I'm trying to write to Datastore, wait for it to complete, then perform a task.
In order to do this, the DatastoreV1.write() transform would need to return
PCollection<Void>. Looking at the code, it returns PDone, and I can't find a
workaround.
I propose adding an option to return PCollection<Void>. If I create a chained
method like this then I think I can avoid breaking backwards compatibility:
{code:java}
DatastoreIO.v1().write().withReturn()
{code}
Alternatively, an entirely new write() method:
{code:java}
DatastoreIO.v1().writeWithReturn(){code}
To illustrate the use case:
{code:java}
PCollection<Entity> collection = someMethodToReturnPCollection();
PCollection<Void> datastoreWrites = collection
.apply(
DatastoreIO.v1().write().withProjectId(projectId)
);
PCollection<Void> otherCollection = ... // get the collection
otherCollection
.apply(
Wait.on(datastoreWrites)
)
.apply(
SomeOtherTransform()
);{code}
I'll be creating a pull request soon to implement this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
