Re: Setting environment variables of the taskmanagers (yarn)

2019-09-25 Thread Peter Huang
Hi Richard, Good suggestion. I just created a Jira ticket. I will find a time this week to update docs. Best Regards Peter Huang On Wed, Sep 25, 2019 at 8:05 AM Richard Deurwaarder wrote: > Hi Peter and Jiayi, > > Thanks for the answers this worked perfectly, I just added > >

Re: Setting environment variables of the taskmanagers (yarn)

2019-09-25 Thread Richard Deurwaarder
Hi Peter and Jiayi, Thanks for the answers this worked perfectly, I just added containerized.master.env.GOOGLE_APPLICATION_CREDENTIALS=xyz and containerized.taskmanager.env.GOOGLE_APPLICATION_CREDENTIALS=xyz to my flink config and they got picked up. Do you know why this is missing from the

Re: Setting environment variables of the taskmanagers (yarn)

2019-09-24 Thread Peter Huang
Hi Richard, For the first question, I don't think you need to explicitly specify fs.hdfs.hadoopconf as each file in the ship folder is copied as a yarn local resource for containers. The configuration path is overridden internally in Flink. For the second question of setting TM environment

Re: Setting environment variables of the taskmanagers (yarn)

2019-09-24 Thread bupt_ljy
Original Message Sender: Richard Deurwaarder Recipient: user Date: Tuesday, Sep 24, 2019 23:01 Subject: Setting environment variables of the taskmanagers (yarn) Hello, We have our flink job (1.8.0) running on our hadoop 2.7 cluster with yarn. We would like to add the GCS connector to use GCS

Setting environment variables of the taskmanagers (yarn)

2019-09-24 Thread Richard Deurwaarder
Hello, We have our flink job (1.8.0) running on our hadoop 2.7 cluster with yarn. We would like to add the GCS connector to use GCS rather than HDFS. Following the documentation of the GCS connector[1] we have to specify which credentials we want to use and there are two ways of doing this: *