Thanks everyone for your suggestions!
From: Ning Kang <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, April 22, 2021 at 10:51 AM
To: "[email protected]" <[email protected]>
Cc: Yuan Feng <[email protected]>
Subject: Re: Any easy way to extract values from PCollection?
+1 to Brian's answer.
In Java, you can
singleValuedPcollection .apply("Write single value",
TextIO.write().to(options.getSomeGcsPath())
as the last step of your pipeline.
Then in your program, after executing the pipeline (wait until finish), use the
Cloud Storage Java client
library<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloud.google.com%2Fstorage%2Fdocs%2Freference%2Flibraries%23client-libraries-install-java&data=04%7C01%7Ctaol%40zillow.com%7C4e6575041bff4296e29b08d905b73d3b%7C033464830d1840e7a5883784ac50e16f%7C0%7C0%7C637547106828526784%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xX2JWh4DYPeJNt9H%2FUS1AMvsvFVZ1aXdcb0pMaybAik%3D&reserved=0>
to read the file and extract the value.
On Thu, Apr 22, 2021 at 10:45 AM Brian Hulette
<[email protected]<mailto:[email protected]>> wrote:
I don't think there's an easy answer to this question, in general all you can
do with a PCollection is indicate you'd like to write it out to an IO. There
has been some work in the Python SDK on "Interactive Beam" which is designed
for using the Python SDK interactively in a notebook environment. It will let
you collect() a PCollection - meaning it runs the pipeline and materializes the
result. There's no such capability for the other SDKs though.
On Wed, Apr 21, 2021 at 8:24 PM Tao Li
<[email protected]<mailto:[email protected]>> wrote:
Hi Beam community,
This is the question I am asking:
https://stackoverflow.com/questions/28015924/how-to-extract-contents-from-pcollection-in-cloud-dataflow<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F28015924%2Fhow-to-extract-contents-from-pcollection-in-cloud-dataflow&data=04%7C01%7Ctaol%40zillow.com%7C4e6575041bff4296e29b08d905b73d3b%7C033464830d1840e7a5883784ac50e16f%7C0%7C0%7C637547106828526784%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cdAouxo9JBRfB%2BNLKllTspBWoYDd7jMU7d2%2F9KvWI8A%3D&reserved=0>
Thanks!
