Re: flink:latest container on kubernetes fails to connect taskmanager to jobmanager

Hi jwatte, sorry for the inconveniences. I hope that the dicker hub images have been updated by now. Cheers, Till On Wed, Oct 10, 2018, 05:20 vino yang wrote: > Hi jwatte, > > Maybe Till can help you. > > Thanks, vino. > > jwatte 于2018年10月2日周二 上午5:30写道： > >> It turns out that the latest

回复：Small checkpoint data takes too much time

The checkpoint duration includes the processes of barrier alignment and state snapshot. Every task has to receive all the barriers from all the channels, then trriger to snapshot state. I guess the barrier alignment may take long time for your case, and it is specially critical during

Small checkpoint data takes too much time

Hi I recently encounter a problem in production. I found checkpoint takes too much time, although it doesn`t affect the job execution. I am using FsStateBackend, writing the data to a HDFS checkpointDataUri, and asynchronousSnapshots, I print the metric data

Re: No data issued by flink window after a few hours

Hi, I saw the exception image you provided. Based on the exception message, it seems you used the default max state size (5MB). You can specify the max state size to override the default value. Try : *MemoryStateBackend stateBackend = new MemoryStateBackend(theSizeOfBytes);* Please note that

Re: No data issued by flink window after a few hours

Hi, Did you mean "computer momery" referring to Memory Statebackend? The Flink window mechanism is internally based on State, and this is done for fault tolerance. If you introduce external storage, it will break its design and bring other problems. Thanks, vino. 潘 功森 于2018年10月10日周三 上午11:02写道：

Re: flink:latest container on kubernetes fails to connect taskmanager to jobmanager

Hi jwatte, Maybe Till can help you. Thanks, vino. jwatte 于2018年10月2日周二 上午5:30写道： > It turns out that the latest flink:latest docker image is 5 days old, and > thus bug was fixed 4 days ago in the flink-docker github. > > The problem is that the docker-entrypoint.sh script chains to

答复: No data issued by flink window after a few hours

Hi, "ram to cache the distinct data about sliding window" means I used computer momery not the third part db to cache the data need used in window. “the data need used in window” means ：such as the sliding window is 1 hour, and I need to count the distinct users, I need to cache the user id

Re: No data issued by flink window after a few hours

Hi, Can you explain what "ram to cache the distinct data about sliding window" mean? The information you provide is too small and will not help others to help you analyze the problem and provide advice. In addition, regarding the usage of Flink related issues, please only send mail to the user

No data issued by flink window after a few hours

Hi all, I used flink window, and when the job begins, we could get the results of windiow.But there’re no results issued after a few hours. I found the job is still running and no errors, and the data not used window all can be issued. By the way, I used Flink 1.3.2 and ram to cache

FlinkKafkaProducer and Confluent Schema Registry

Hello, I would like to use Confluent Schema Registry in my streaming job. I was able to make it work with the help of generic Kafka producer and FlinkKafkaConsumer which is using ConfluentRegistryAvroDeserializationSchema. FlinkKafkaConsumer011 consumer = new FlinkKafkaConsumer011<>(MY_TOPIC,

[DISCUSS] Integrate Flink SQL well with Hive ecosystem

Hi all, Along with the community's effort, inside Alibaba we have explored Flink's potential as an execution engine not just for stream processing but also for batch processing. We are encouraged by our findings and have initiated our effort to make Flink's SQL capabilities full-fledged. When

Re: JobManager did not respond within 60000 ms

Hi Piotrek, Thank you for your answer. Actually it was necessary to increase the memory of the JobManager (I had tested it but I had not restarted Flink ...). I will also work on optimization. I thought it was good practice to create as much function as possible based on their functional value

Re: Flink 1.5.2 - excessive ammount of container requests, Received new/Returning excess container "flood"

Hey guys, thanks for the replies. 1. "Requesting new TaskExecutor" looks fine as it's exactly 32 as is jobs' parallelism set. The weird thing is that after those 32 containers requested and received we have this "flood" of 'Received new container/Returning excess container` (and as shown below

getRuntimeContext(): The runtime context has not been initialized.

Hi, We want to use MapState inside fold function to keep the map of all products that we see in 24 hour window to store huge state in rocksdb rather than overflowing heap. However, I don't seem to initialise mapstate within foldfunction or any class that is extending RichMapFunction private

Re: BroadcastStream vs Broadcasted DataStream

Hi Pieter-Jan, The second variant stores the elements of the broadcasted stream in operator (thus non-keyed) state. On the differences: The Broadcast stream is not a keyed stream, so you are not in a keyed context, thus you have no access to keyed state. Given this, and assuming that you are

[ANNOUNCE] Weekly community update #41

Dear community, this is the weekly community update thread #41. Please post any news and updates you want to share with the community to this thread. # Feature freeze for Flink 1.7 The community has decided to freeze the feature development for Flink 1.7.0 on the 22nd of October [1]. # Flink

Re: Ship compiled code with broadcast stream ?

You should not try sending the compiled code anywhere but you can use it from within the processor. You can do the same thing with the jar, you compile your jar, store it on HDFS. Send the jar path to the processor which can download the jar and instantiate the rule. No need to resubmit the job.

Re: Flink leaves a lot RocksDB sst files in tmp directory

After digging more in the log, I think it's more a bug. I've greped a log by job id and found under normal circumstances TM supposed to delete flink-io files. For some reason, it doesn't delete files that were listed above. 2018-10-08 22:10:25,865 INFO

Re: Ship compiled code with broadcast stream ?

Hi, This is certainly possible. What you can do is use a > BroadcastProcessFunction where you receive the rule code on the broadcast > side. > Yes, this part works, no problem. > You probably cannot send newly compiled objects this way but what you can > do is either send a reference to some

BroadcastStream vs Broadcasted DataStream

Hi, I am not sure I fully understand the differences between doing something like dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{}) and this dataStreamX.connect(dataStreamY.broadcast(*descriptor*).process(new BroadcastProcessFunction) Couldn't I manage the state

Re: Wondering how to check if taskmanager is registered with the jobmanager, without asking job manager

Version of flink I'm using is 1.6.1, if that helps. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Wondering how to check if taskmanager is registered with the jobmanager, without asking job manager

As part of deploying task managers and job managers, I'd like to expose healthcheck on both task managers and job managers. For the task managers, one of the requirements that they are healthy, is that they have successfully registered themselves with the job manager. Is there a way to achieve

Re: Duplicates in self join

Interval join is exactly what I'm looking for. Thanks for pointing it out! On Mon, Oct 8, 2018 at 9:13 AM Fabian Hueske wrote: > Did you check the new interval join that was added with Flink 1.6.0 [1]? > It might be better suited because, each record has its own boundaries > based on its

Re: JobManager did not respond within 60000 ms

Hi, You have quite complicated job graph and very low memory settings for the job manager and task manager. It might be that long GC pauses are causing this problem. Secondly, there are quite some results in google search

Re: [DISCUSS] Dropping flink-storm?

Yes, let's do it this way. The wrapper classes are probably not too complex and can be easily tested. We have the same for the Hadoop interfaces, although I think only the Input- and OutputFormatWrappers are actually used. Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler <

Re: Ship compiled code with broadcast stream ?

Hi, This is certainly possible. What you can do is use a BroadcastProcessFunction where you receive the rule code on the broadcast side. You probably cannot send newly compiled objects this way but what you can do is either send a reference to some compiled jars and load them with the

JobManager did not respond within 60000 ms

I have a streaming job that works in standalone cluster. Flink version is 1.4.1. Everything was working so far. But since I added new treatments, I can not start my job anymore. I have this exception : org.apache.flink.client.program.ProgramInvocationException: The program execution failed:

Re: [DISCUSS] Dropping flink-storm?

That sounds very good to me. On 08.10.2018 11:36, Till Rohrmann wrote: Good point. The initial idea of this thread was to remove the storm compatibility layer completely. During the discussion I realized that it might be useful for our users to not completely remove it in one go. Instead for

Ship compiled code with broadcast stream ?

Hi everyone, I have a job with event stream and control stream delivering rules for event transformation. Rules are broadcasted and used in flatMat-like coProcessFunction. Rules are defined in custom JSON format. Amount of rules and complexity rises significantly with every new feature. What I

Re: Duplicates in self join

When I switched to using TumblingEventTimeWindows, it did remove the duplicates, which was somewhat surprising because with just 10 edges (.1 seconds in length), it should have fit within one window of the SlidingEventTimeWindows (20 seconds window, 10 second slide). On Mon, Oct 8, 2018 at 9:02

Re: HDFS HA issue in Flink 1.4 caused by failing to use fs.hdfs.hadoopconf in Flink conf

Thanks for figuring this out, Shuyi! > On 9. Oct 2018, at 09:09, Shuyi Chen wrote: > > I think the bug is introduced in FLINK-7643 (Rework FileSystem loading to use > factories). In YarnApplicationMasterRunner, after the JIRA, FileSystem was > not properly initialized with the correct flink

Re: HDFS HA issue in Flink 1.4 caused by failing to use fs.hdfs.hadoopconf in Flink conf

I think the bug is introduced in FLINK-7643 (Rework FileSystem loading to use factories). In YarnApplicationMasterRunner, after the JIRA, FileSystem was not properly initialized with the correct flink configuration before calling runApplicationMaster(). W/o the initialization, a call of

Re: Watermark through Rest Api

Hi, Watermarks are tracked per Task/Operator level: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#io Tracking watermarks on the job level would be problematic, since it would