RE: Flink on Mesos: containers question

PM To: Till Rohrmann Cc: NEKRASSOV, ALEXEI ; Fabian Hueske ; user Subject: Re: Flink on Mesos: containers question Hi， Alexei: What you paste is expected behavior. Jobmanager, two task managers each should run in a docker instance. 13276 is should be the process of job manager, and it's

RE: Flink on Mesos: containers question

Till, Any insight into how Flink components are containerized in Mesos? Thanks! Alex From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: Monday, July 16, 2018 7:57 AM To: NEKRASSOV, ALEXEI Cc: user@flink.apache.org; Till Rohrmann Subject: Re: Flink on Mesos: containers question Hi Alexei

Flink on Mesos: containers question

Can someone please clarify how Flink on Mesos in containerized? On 5-node Mesos cluster I started Flink (1.4.2) with two Task Managers. Mesos shows "flink" task and two "taskmanager" tasks, all on the same VM. On that VM I see one Docker container running a process that seems to be Mesos App

Checkpointing when reading from files?

I want to add checkpointing to my program that reads from a set of files in a directory. Without checkpointing I use readFile(): DataStream text = env.readFile( new TextInputFormat(new Path(inputPath)), inputPath,

Consolidated log for a job?

Is there a way to see logs from multiple Task Managers *all in one place* (for a running or a completed job)? Or I need to check logs on each Task Manager individually? Thanks, Alex Nekrassov

RE: RocksDBMapState example?

. I'm not clear how ValueState can be used to store key-value mapping. Can you please clarify? Thanks, Alex -Original Message- From: Dawid Wysakowicz [mailto:wysakowicz.da...@gmail.com] Sent: Tuesday, April 10, 2018 8:54 AM To: NEKRASSOV, ALEXEI <an4...@att.com> Cc: user@flin

RE: RocksDBMapState example?

I looked at that code, but I’m still not clear. new RocksDBMapState<>(columnFamily, namespaceSerializer, stateDesc, this); columnFamily is determined by 50-line function; is this necessary for a simple use case like mine? What should I use as state descriptor in that function?.. Last argument

RocksDBMapState example?

Hi, I'd like to use RocksDB to store a key-value mapping table (with 45 million keys). Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?.. Thanks, Alex Nekrassov nekras...@att.com

RE: How does setMaxParallelism work

Is there an auto-scaling feature in Flink, where I start with parallelism of (for example) 1, but Flink notices I have high volume of data to process, and automatically increases parallelism of a running job? Thanks, Alex -Original Message- From: Nico Kruber

timeWindow emits records before window ends?

Hello, With time characteristic set to IngestionTime I expected "timeWindow(Time.minutes(3))" to NOT produce any records in the first 3 minutes of running the job, and yet it does emit the record before 3 minutes elapse. Am I doing something wrong? Or my understanding of timeWindow is

POJO default constructor - how is it used by Flink?

Hi, I added a default constructor to the class that represents messages flowing through my Flink job graph - to satisfy Flink POJO requirements. Although I don't call that default constructor explicitly, the logs show that it is called anyway. Why is this happening? In my test, for 77