Re: beam.io.ReadStringsFromPubSub Unexpected behaviour

[Bayu Aldi Yansyah]

Hmm I got a chance to test my code again and it worked perfectly without
any changes.

Is there any thoughts on why this is happen? So I can make sure that this
is not happen again in the future.

Thanks
— bay

On Fri, 5 Jan 2018 at 14.24 Bayu Aldi Yansyah 
wrote:

> Hey guys,
>
> I use the latest version apache beam package and found a strange behaviour.
> I'm not sure this is bug or not, because 2 hours ago I ran the same code
> and it's fine.
> Now it's error.
>
> If I use a topic name as the parameter:
>
> events = (pipeline
> | 'Read' >> ReadStringsFromPubSub(
> topic=FQ_TOPIC_NAME,
> id_label='event_id')
>
> It runs as expected:
> - It create a subscription: like
> projects/PROJECT_ID/subscriptions/beam_1515134413_fbbb1a48
> - Pull the events
>
> But If I use the subscription as the parameter:
>
> events = (pipeline
> | 'Read' >> ReadStringsFromPubSub(
> subscription=FQ_SUBSCRIPTION,
> id_label='event_id')
> it raise the following exception:
>
> google.gax.errors.RetryError: RetryError(Exception occurred in
> retry method that was not classified as transient, caused by <_Rendezvous
> of RPC that terminated with (StatusCode.UNAUTHENTICATED, Deadline
> Exceeded)>)
>
>
> If I use the random topic name or subscription url, it will throw a
> StatusCode.PERMISSION_DENIED
> which is expected.
>
> If i upgrade the pubsub client pip install --upgrade google-cloud-pubsub
> it will throw an error like the following:
>
> class Field(six.with_metaclass(_FieldMeta, object)):
> TypeError: Error when calling the metaclass bases
> metaclass conflict: the metaclass of a derived class must be a
> (non-strict) subclass of the metaclasses of all its bases
>
>
> Below is my pip freeze output:
>
> apache-beam==2.2.0
> avro==1.8.2
> cachetools==2.0.1
> certifi==2017.11.5
> chardet==3.0.4
> crcmod==1.7
> dill==0.2.6
> enum34==1.1.6
> funcsigs==1.0.2
> future==0.16.0
> futures==3.2.0
> gapic-google-cloud-pubsub-v1==0.15.4
> google-apitools==0.5.11
> google-auth==1.2.1
> google-auth-httplib2==0.0.3
> google-cloud-bigquery==0.25.0
> google-cloud-core==0.25.0
> google-cloud-pubsub==0.26.0
> google-gax==0.15.16
> googleapis-common-protos==1.5.3
> googledatastore==7.0.1
> grpc-google-iam-v1==0.11.4
> grpcio==1.8.3
> httplib2==0.9.2
> idna==2.6
> mock==2.0.0
> oauth2client==3.0.0
> pbr==3.1.1
> ply==3.8
> proto-google-cloud-datastore-v1==0.90.4
> proto-google-cloud-pubsub-v1==0.15.4
> protobuf==3.3.0
> pyasn1==0.4.2
> pyasn1-modules==0.2.1
> PyYAML==3.12
> requests==2.18.4
> rsa==3.4.2
> six==1.10.0
> typing==3.6.2
> urllib3==1.22
>
> What should I do?
> Thanks
>
-- 
Bayu Aldi Yansyah,
Universitas Airlangga
https://github.com/pyk

Re: Regarding Beam Slack Channel

[Shashank Prabhakara]

Got it, thanks.

On 9 Jan 2018 11:53 am, "Jean-Baptiste Onofré"  wrote:

> Done, you should have received the invite.
>
> Welcome !
>
> Regards
> JB
>
> On 01/09/2018 06:31 AM, Shashank Prabhakara wrote:
>
>> I'd also like to be added, please.
>>
>> Thanks.
>>
>> On 2018-01-04 11:58, Jean-Baptiste Onofr�  j...@nanthrax.net>> wrote:
>>  > Hi,>
>>  >
>>  > you should have received the invite.>
>>  >
>>  > Welcome aboard !>
>>  >
>>  > Regards>
>>  > JB>
>>  >
>>  > On 01/04/2018 01:38 AM, David Sabater Dinter wrote:>
>>  > > Hi,>
>>  > > Can I join also, please?>
>>  > > >
>>  > > Thanks!>
>>  > > >
>>  > > On Thu, Jan 4, 2018 at 12:54 AM Maria Tjahjadi <
>> maria.tjahj...@tokopedia.com  >
>>  > > >> wrote:>
>>  > > >
>>  > > Hi,>
>>  > > >
>>  > > Could I join also?>
>>  > > >
>>  > > Thanks,>
>>  > > Maria Tjahjadi>
>>  > > >
>>  > > On 4 Jan 2018, at 06.15, Lukasz Cwik  lc...@google.com>>
>>  > > >> wrote:>
>>  > > >
>>  > >> Invite sent, welcome.>
>>  > >>>
>>  > >> On Wed, Jan 3, 2018 at 3:01 PM, Carlos Alonso <
>> car...@mrcalonso.com >
>>  > >> >> wrote:>
>>  > >>>
>>  > >> Hi, I'd like to be added too, please!>
>>  > >>>
>>  > >> Thanks!>
>>  > >>>
>>  > >> On Tue, Dec 19, 2017 at 3:41 PM Jean-Baptiste Onofr� <
>> j...@nanthrax.net >
>>  > >> >> wrote:>
>>  > >>>
>>  > >> Done,>
>>  > >>>
>>  > >> you should have received an invite.>
>>  > >>>
>>  > >> Regards>
>>  > >> JB>
>>  > >>>
>>  > >> On 12/19/2017 03:20 PM, Unais T wrote:>
>>  > >> > Hello>
>>  > >> >>
>>  > >> > Can someone please add me to the Beam slack channel?>
>>  > >> >>
>>  > >> > Thanks.>
>>  > >> >>
>>  > >> >>
>>  > >>>
>>  > >> -->
>>  > >> Jean-Baptiste Onofr�>
>>  > >> jbono...@apache.org  > >>
>>  > >> http://blog.nanthrax.net>
>>  > >> Talend - http://www.talend.com>
>>  > >>>
>>  > >>>
>>  >
>>  > -- >
>>  > Jean-Baptiste Onofr�>
>>  > jbono...@apache.org >
>>  > http://blog.nanthrax.net>
>>  > Talend - http://www.talend.com>
>>  >
>>
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Regarding Beam Slack Channel

[Jean-Baptiste Onofré]

Done, you should have received the invite.

Welcome !

Regards
JB

On 01/09/2018 06:31 AM, Shashank Prabhakara wrote:

I'd also like to be added, please.

Thanks.

On 2018-01-04 11:58, Jean-Baptiste Onofr� > wrote:

 > Hi,>
 >
 > you should have received the invite.>
 >
 > Welcome aboard !>
 >
 > Regards>
 > JB>
 >
 > On 01/04/2018 01:38 AM, David Sabater Dinter wrote:>
 > > Hi,>
 > > Can I join also, please?>
 > > >
 > > Thanks!>
 > > >
 > > On Thu, Jan 4, 2018 at 12:54 AM Maria Tjahjadi 
 >

 > > >> wrote:>
 > > >
 > >     Hi,>
 > > >
 > >     Could I join also?>
 > > >
 > >     Thanks,>
 > >     Maria Tjahjadi>
 > > >
 > >     On 4 Jan 2018, at 06.15, Lukasz Cwik >

 > >     >> wrote:>
 > > >
 > >>     Invite sent, welcome.>
 > >>>
 > >>     On Wed, Jan 3, 2018 at 3:01 PM, Carlos Alonso >

 > >>     >> wrote:>
 > >>>
 > >>         Hi, I'd like to be added too, please!>
 > >>>
 > >>         Thanks!>
 > >>>
 > >>         On Tue, Dec 19, 2017 at 3:41 PM Jean-Baptiste Onofr� 
>

 > >>         >> wrote:>
 > >>>
 > >>             Done,>
 > >>>
 > >>             you should have received an invite.>
 > >>>
 > >>             Regards>
 > >>             JB>
 > >>>
 > >>             On 12/19/2017 03:20 PM, Unais T wrote:>
 > >>             > Hello>
 > >>             >>
 > >>             > Can someone please add me to the Beam slack channel?>
 > >>             >>
 > >>             > Thanks.>
 > >>             >>
 > >>             >>
 > >>>
 > >>             -->
 > >>             Jean-Baptiste Onofr�>
 > >> jbono...@apache.org  >>

 > >> http://blog.nanthrax.net>
 > >>             Talend - http://www.talend.com>
 > >>>
 > >>>
 >
 > -- >
 > Jean-Baptiste Onofr�>
 > jbono...@apache.org >
 > http://blog.nanthrax.net>
 > Talend - http://www.talend.com>
 >


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Regarding Beam Slack Channel

[Shashank Prabhakara]

Hi Austin,

Not sure what you meant by follow along there, but I can't sign up on the
slack page due to email domain restrictions and it does not allow me to
request entry there.

Regards,
Shashank

On Tue, Jan 9, 2018 at 11:11 AM, Austin Bennett  wrote:

> It'd be easier to follow along there
>
> On Jan 8, 2018 9:32 PM, "Shashank Prabhakara" 
> wrote:
>
>> I'd also like to be added, please.
>>
>> Thanks.
>>
>> On 2018-01-04 11:58, Jean-Baptiste Onofr�  wrote:
>> > Hi,>
>> >
>> > you should have received the invite.>
>> >
>> > Welcome aboard !>
>> >
>> > Regards>
>> > JB>
>> >
>> > On 01/04/2018 01:38 AM, David Sabater Dinter wrote:>
>> > > Hi,>
>> > > Can I join also, please?>
>> > > >
>> > > Thanks!>
>> > > >
>> > > On Thu, Jan 4, 2018 at 12:54 AM Maria Tjahjadi <
>> maria.tjahj...@tokopedia.com >
>> > > > wrote:>
>> > > >
>> > > Hi,>
>> > > >
>> > > Could I join also?>
>> > > >
>> > > Thanks,>
>> > > Maria Tjahjadi>
>> > > >
>> > > On 4 Jan 2018, at 06.15, Lukasz Cwik 
>> > > > wrote:>
>> > > >
>> > >> Invite sent, welcome.>
>> > >>>
>> > >> On Wed, Jan 3, 2018 at 3:01 PM, Carlos Alonso <
>> car...@mrcalonso.com>
>> > >> > wrote:>
>> > >>>
>> > >> Hi, I'd like to be added too, please!>
>> > >>>
>> > >> Thanks!>
>> > >>>
>> > >> On Tue, Dec 19, 2017 at 3:41 PM Jean-Baptiste Onofr� <
>> j...@nanthrax.net>
>> > >> > wrote:>
>> > >>>
>> > >> Done,>
>> > >>>
>> > >> you should have received an invite.>
>> > >>>
>> > >> Regards>
>> > >> JB>
>> > >>>
>> > >> On 12/19/2017 03:20 PM, Unais T wrote:>
>> > >> > Hello>
>> > >> >>
>> > >> > Can someone please add me to the Beam slack channel?>
>> > >> >>
>> > >> > Thanks.>
>> > >> >>
>> > >> >>
>> > >>>
>> > >> -->
>> > >> Jean-Baptiste Onofr�>
>> > >> jbono...@apache.org >
>> > >> http://blog.nanthrax.net>
>> > >> Talend - http://www.talend.com>
>> > >>>
>> > >>>
>> >
>> > -- >
>> > Jean-Baptiste Onofr�>
>> > jbono...@apache.org>
>> > http://blog.nanthrax.net>
>> > Talend - http://www.talend.com>
>> >
>>
>

Re: Regarding Beam Slack Channel

[Austin Bennett]

It'd be easier to follow along there

On Jan 8, 2018 9:32 PM, "Shashank Prabhakara"  wrote:

> I'd also like to be added, please.
>
> Thanks.
>
> On 2018-01-04 11:58, Jean-Baptiste Onofr�  wrote:
> > Hi,>
> >
> > you should have received the invite.>
> >
> > Welcome aboard !>
> >
> > Regards>
> > JB>
> >
> > On 01/04/2018 01:38 AM, David Sabater Dinter wrote:>
> > > Hi,>
> > > Can I join also, please?>
> > > >
> > > Thanks!>
> > > >
> > > On Thu, Jan 4, 2018 at 12:54 AM Maria Tjahjadi <
> maria.tjahj...@tokopedia.com >
> > > > wrote:>
> > > >
> > > Hi,>
> > > >
> > > Could I join also?>
> > > >
> > > Thanks,>
> > > Maria Tjahjadi>
> > > >
> > > On 4 Jan 2018, at 06.15, Lukasz Cwik 
> > > > wrote:>
> > > >
> > >> Invite sent, welcome.>
> > >>>
> > >> On Wed, Jan 3, 2018 at 3:01 PM, Carlos Alonso <
> car...@mrcalonso.com>
> > >> > wrote:>
> > >>>
> > >> Hi, I'd like to be added too, please!>
> > >>>
> > >> Thanks!>
> > >>>
> > >> On Tue, Dec 19, 2017 at 3:41 PM Jean-Baptiste Onofr� <
> j...@nanthrax.net>
> > >> > wrote:>
> > >>>
> > >> Done,>
> > >>>
> > >> you should have received an invite.>
> > >>>
> > >> Regards>
> > >> JB>
> > >>>
> > >> On 12/19/2017 03:20 PM, Unais T wrote:>
> > >> > Hello>
> > >> >>
> > >> > Can someone please add me to the Beam slack channel?>
> > >> >>
> > >> > Thanks.>
> > >> >>
> > >> >>
> > >>>
> > >> -->
> > >> Jean-Baptiste Onofr�>
> > >> jbono...@apache.org >
> > >> http://blog.nanthrax.net>
> > >> Talend - http://www.talend.com>
> > >>>
> > >>>
> >
> > -- >
> > Jean-Baptiste Onofr�>
> > jbono...@apache.org>
> > http://blog.nanthrax.net>
> > Talend - http://www.talend.com>
> >
>

Re: Regarding Beam Slack Channel

[Shashank Prabhakara]

I'd also like to be added, please.

Thanks.

On 2018-01-04 11:58, Jean-Baptiste Onofr�  wrote:
> Hi,>
>
> you should have received the invite.>
>
> Welcome aboard !>
>
> Regards>
> JB>
>
> On 01/04/2018 01:38 AM, David Sabater Dinter wrote:>
> > Hi,>
> > Can I join also, please?>
> > >
> > Thanks!>
> > >
> > On Thu, Jan 4, 2018 at 12:54 AM Maria Tjahjadi <
maria.tjahj...@tokopedia.com >
> > > wrote:>
> > >
> > Hi,>
> > >
> > Could I join also?>
> > >
> > Thanks,>
> > Maria Tjahjadi>
> > >
> > On 4 Jan 2018, at 06.15, Lukasz Cwik 
> > > wrote:>
> > >
> >> Invite sent, welcome.>
> >>>
> >> On Wed, Jan 3, 2018 at 3:01 PM, Carlos Alonso 

> >> > wrote:>
> >>>
> >> Hi, I'd like to be added too, please!>
> >>>
> >> Thanks!>
> >>>
> >> On Tue, Dec 19, 2017 at 3:41 PM Jean-Baptiste Onofr� <
j...@nanthrax.net>
> >> > wrote:>
> >>>
> >> Done,>
> >>>
> >> you should have received an invite.>
> >>>
> >> Regards>
> >> JB>
> >>>
> >> On 12/19/2017 03:20 PM, Unais T wrote:>
> >> > Hello>
> >> >>
> >> > Can someone please add me to the Beam slack channel?>
> >> >>
> >> > Thanks.>
> >> >>
> >> >>
> >>>
> >> -->
> >> Jean-Baptiste Onofr�>
> >> jbono...@apache.org >
> >> http://blog.nanthrax.net>
> >> Talend - http://www.talend.com>
> >>>
> >>>
>
> -- >
> Jean-Baptiste Onofr�>
> jbono...@apache.org>
> http://blog.nanthrax.net>
> Talend - http://www.talend.com>
>

Re: London Apache Beam meetup 2: 11/01

[Matthias Baetens]

Yes, we put everything in place to record this time and hope to share the
recordings soon after the meetup. Stay tuned!

On 8 Jan 2018 10:32, "Carlos Alonso"  wrote:

Will it be recorded?

On Fri, Jan 5, 2018 at 5:11 PM Matthias Baetens  wrote:

> Hi all,
>
> Excited to announce the second Beam meet up located in the *Qubit offices
>  *next *Thursday 11/01.*
>
> We are very excited to have JB flying in to talk about IO and Splittable
> DoFns and Vadim Sobolevski to share on how FutureFlow uses Beam in a
> finance use case.
>
> More info and RSVP here . We are looking forward
> to welcome you all!
>
> Best regards,
> Matthias
>

Re: Need help running beam word count example on apex/hdfs

[Shashank Prabhakara]

Forgot to mention:

Execution works in embedded mode and counts are created on the local fs. I
need this to run on hdfs/yarn with --embeddedExecution=false.

Regards,
Shashank

On Mon, Jan 8, 2018 at 3:06 PM, Shashank Prabhakara 
wrote:

> Hi All,
>
> I want to test beam on apex using the word count example provided in the
> beam repository, but I'm facing some difficulties while executing word
> count as described in the documentation.
>
> I'm running hadoop version 2.8.2 on debian in a multi-node environment.
> I cloned the beam github repository - master branch and executed:
>
> cd examples/java
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount
> -Dexec.args="--inputFile=/tmp/input/pom.xml --output=/tmp/output/counts
> --runner=ApexRunner --embeddedExecution=false" -Papex-runner
>
> However the driver hangs (waited for > 1hr) after printing the classpath
> on the console. I have attached the stdout and stacktrace to this (Pls let
> me know if not visible in ML).
>
> Thanks in advance for any help.
>
> Regards,
> Shashank
>

Re: London Apache Beam meetup 2: 11/01

[Carlos Alonso]

Will it be recorded?

On Fri, Jan 5, 2018 at 5:11 PM Matthias Baetens <
matthias.baet...@datatonic.com> wrote:

> Hi all,
>
> Excited to announce the second Beam meet up located in the *Qubit offices
>  *next *Thursday 11/01.*
>
> We are very excited to have JB flying in to talk about IO and Splittable
> DoFns and Vadim Sobolevski to share on how FutureFlow uses Beam in a
> finance use case.
>
> More info and RSVP here . We are looking forward
> to welcome you all!
>
> Best regards,
> Matthias
>

Need help running beam word count example on apex/hdfs

[Shashank Prabhakara]

Hi All,

I want to test beam on apex using the word count example provided in the
beam repository, but I'm facing some difficulties while executing word
count as described in the documentation.

I'm running hadoop version 2.8.2 on debian in a multi-node environment.
I cloned the beam github repository - master branch and executed:

cd examples/java
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount
-Dexec.args="--inputFile=/tmp/input/pom.xml --output=/tmp/output/counts
--runner=ApexRunner --embeddedExecution=false" -Papex-runner

However the driver hangs (waited for > 1hr) after printing the classpath on
the console. I have attached the stdout and stacktrace to this (Pls let me
know if not visible in ML).

Thanks in advance for any help.

Regards,
Shashank


stacktrace.out
Description: Binary data


std.out
Description: Binary data