Re: Setting number of TaskManagers

Hi Craig, For the YARN session, you have to pass the the number of taskManagers using the -n argument. if you need to use a n environment variable, you can create a custom script calling the yarn-session.sh script and passing the value of the env variable to the script. Regards, Robert On

Re: Flink long-running YARN configuration

Hi Craig! For YARN sessions, Flink will - (a) register the app master hostname/port/etc at Yarn, so you can get them from example from the yarn UI and tools - (b) it will create a .yarn-properties file that contain the hostname/ports info. Future calls to the command line pick up the info

Re: Setting up zeppelin with flink

I'm glad you were able to work it out! Your setup is somewhat unique, and as Zeppelin is the result of multiple drive-by commits, interesting and unexpected things happen in the tail cases. Could you please report your problem and solution on the Zeppelin user list? What you've discovered may

Re: Running multiple jobs on yarn (without yarn-session)

Thanks Niels, actually I also created one :) We will fix this on the master and for the 1.1.2 release. On Thu, Aug 25, 2016 at 5:14 PM, Niels Basjes wrote: > I have this with a pretty recent version of the source version (not a > release). > > Would be great if you see a way to

Re: Elasticsearch connector and number of shards

I've just added a JIRA improvement ticket for this ( https://issues.apache.org/jira/browse/FLINK-4491). Best, Flavio On Wed, Jul 20, 2016 at 4:21 PM, Maximilian Michels wrote: > The connector doesn't cover this use case. Through the API you need to > use the

Re: Dealing with Multiple sinks in Flink

Hi Max, Here is the code for Timestamp assigner and watermark generation. PFA Regards, Vinay Patil On Thu, Aug 25, 2016 at 7:39 AM, Maximilian Michels [via Apache Flink User Mailing List archive.] wrote: > I'm assuming there is something wrong with your

Re: Setting up zeppelin with flink

Hello, Sorry for the spam, but i got it working after copying all scala libraries from another interpreter to the interpreter/flink directory. so i think the error is the scala libraries are missing from the binary release in the zeppelin/interpreters/flink/ directory. For now i'm adding the copy

Re: Running multiple jobs on yarn (without yarn-session)

Hi Niels, This is with 1.1.1? We could fix this in the upcoming 1.1.2 release by only using automatic shut down for detached jobs. In all other cases we should be able to shutdown from the client side after running all jobs. The only downside I see is that Flink clusters may actually never be

Re: Setting up zeppelin with flink

Hello, For reference, below is the dockerfile i used to build the zeppelin image (basically just openjdk 8 with the latest binary release of zeppelin) the "docker-entrypoint.sh" script is just starting zeppelin.sh (oneliner) FROM openjdk:alpine RUN apk add --no-cache bash snappy ARG

Re: Setting up zeppelin with flink

Hello Trevor, Thanks for your suggestion. The log does not explain a lot: on the flink side i don't see anything at all, on the zeppelin side i see this: Your suggestion sounds plausible, as i always start zeppelin, and then change the configuration from local to remote.. however, port 6123

Re: Delaying starting the jobmanager in yarn?

Hi Niels, If you're using 1.1.1, then you can instantiate the YarnClusterDescriptor and supply it with the Flink jar and configuration and subsequently call `deploy()` on it to receive a ClusterClient for Yarn which you can submit programs using the `run(PackagedProgram program, String args)`

Re: "Failed to retrieve JobManager address" in Flink 1.1.1 with JM HA

Max, Thank you for the fix! Regards, Hironori 2016-08-24 18:37 GMT+09:00 Maximilian Michels : > Hi Hironori, > > That's what I thought. So it won't be an issue for most users who do > not comment out the JobManager url from the config. Still, the > information printed is not

Re: Dealing with Multiple sinks in Flink

I'm assuming there is something wrong with your Watermark/Timestamp assigner. Could you share some of the code? On Wed, Aug 24, 2016 at 9:54 PM, vinay patil wrote: > Hi, > > Just an update, the window is not getting triggered when I change the > parallelism to more than

Re: Regarding Global Configuration in Flink

Hi! Are you referring to the GlobalConfiguration class? That used to be a singleton class in Flink version < 1.2.x which would load the configuration only once per VM, if it found a config file. It allowed operations that could change that config after it had been loaded. It has since then been

Running multiple jobs on yarn (without yarn-session)

Hi, I created a small application that needs to run multiple (batch) jobs on Yarn and then terminate. In this case I'm exporting data from a list of HBase tables I essentially do right now the following: flink run -m yarn-cluster -yn 10 bla.jar ... And in my main I do foreach thing I need to

Re: How to set a custom JAVA_HOME when run flink on YARN?

Preferably, you set that directly in the config using env.java.home: /path/to/java/home If unset, Flink will use the $JAVA_HOME environment variable. Cheers, Max On Thu, Aug 25, 2016 at 10:39 AM, Renkai wrote: > I think I solved myself,just add -yD

Re: How to set a custom JAVA_HOME when run flink on YARN?

I think I solved myself,just add -yD yarn.taskmanager.env.JAVA_HOME=xx in the command line, a little hard to find the solution though. -- View this message in context:

Delaying starting the jobmanager in yarn?

Hi, We have a situation where we need to start a flink batch job on a yarn cluster the moment an event arrives over a queue. These events occur at a very low rate (like once or twice a week). The idea we have is to run an application that listens to the queue and executes the batch when it

Re: flink - Working with State example

Hi, you mean the directory is completely empty? Can you check in the JobManager dashboard whether it reports any successful checkpoints for the job? One possible explanation is an optimization that the FsStateBackend performs: when the state is very small it will not actually be written to files