Re: Reduce parallelism without network transfer.

Hi, It should work like this out of the box if you use rescale method: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html#physical-partitioning

Re: Flink not writing last few elements to disk

Hi, FileProcessMode.PROCESS_CONTINUOUSLY processes the file continuously - the stream will not end. Simple `writeAsCsv(…)` on the other hand only flushes the output file on a stream end (see `OutputFormatSinkFunction`). You can either use `PROCESS_ONCE` mode or use more advanced data sink: -

Re: Latest version of Kafka

Hi, Flink as for now provides only a connector for Kafka 0.11, which is using KafkaClient in 0.11.x version. However you should be able to use it for reading to/writing from Kafka 1.0 - Kafka claims (and as far as I know it’s true) that Kafka 1.0 is backward compatible with 0.11. Piotrek >

Re: Send ACK when all records of file are processed

the > LONG.MAX_VALUE, it will get complicated. > > > > Regards, > Vinay Patil > > On Tue, Jan 30, 2018 at 1:57 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Thanks for the clarification :) > > Since you hav

Re: Send ACK when all records of file are processed

.addSink(// DB) > > instead of add sink, should it be a simple map operator which writes to DB > so that we can have a next ack operator which will generate the response. > > Also, how do I get/access the Watermark value in the ack operator ? It will > be a simple map

Re: Data exchange between tasks (operators/sources) at streaming api runtime

> Thanks for the pointers. > Ishwara Varnasi > > Sent from my iPhone > > On Jan 25, 2018, at 4:03 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > >> Hi, >> >> As far as I know there is currently

Re: Send ACK when all records of file are processed

Hi, As you figured out, some dummy EOF record is one solution, however you might try to achieve it also by wrapping an existing CSV function. Your wrapper could emit this dummy EOF record. Another (probably better) idea is to use Watermark(Long.MAX_VALUE) for the EOF marker. Stream source

Re: Avoiding deadlock with iterations

Hi, This is a known problem and I don’t think there is an easy solution to this. Please refer to the: http://mail-archives.apache.org/mod_mbox/flink-user/201704.mbox/%3c5486a7fd-41c3-4131-5100-272825088...@gaborhermann.com%3E

Re: Data exchange between tasks (operators/sources) at streaming api runtime

Hi, As far as I know there is currently no simple way to do this: Join stream with static data in https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API and

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

Hi, > > I attached log file, > > Thanks > > On Mon, Jan 15, 2018 at 3:36 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Could you post full job manager and task manager logs from startup until the &g

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

ce.InstanceManager - Registered > TaskManager at ??? (akka://flink/deadLetters) as > 626846ae27a833cb094eeeb047a6a72c. Current number of registered hosts is 2. > Current number of alive task slots is 40. > > > On Wed, Jan 10, 2018 at 11:36 AM, Piotr Nowojski <pi...@data-artisans.com >

Re: Stream job failed after increasing number retained checkpoints

ns.com>> wrote: > Hi, > > there is no known limitation in the strict sense, but you might run out of > dfs space or job manager memory if you keep around a huge number checkpoints. > I wonder what reason you might have that you ever want such a huge number of > retaine

Re: Datastream broadcast with KeyBy

Hi, Could you elaborate what is the problem that you are having? What is the exception(s) that you are getting? I have tested such simple example and it’s seems to be working as expected: DataStreamSource input = env.fromElements(1, 2, 3, 4, 5, 1, 2, 3); DataStreamSource confStream =

Re: Custom Partitioning for Keyed Streams

Hi, I don’t think it is possible to enforce scheduling of two keys to different nodes, since all of that is based on hashes. For some cases, doing the pre-aggregation step (initial aggregation done before keyBy, which is followed by final aggregation after the keyBy) can be the solution for

Re: Stream job failed after increasing number retained checkpoints

Hi, Increasing akka’s timeouts is rarely a solution for any problems - it either do not help, or just mask the issue making it less visible. But yes, it is possible to bump the limits:

Re: periodic trigger

nt to > discard events from different users received later then i might receive more > than one row for a single user based on the number of windows created by the > events of this user. That will make the the average computations wrong. > > On 22.12.2017 12:10, Piotr Nowojski wrote: >

Re: entrypoint for executing job in task manager

; before any operator opening. > > On Thu, Dec 21, 2017 at 12:27 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Open method is called just before any elements are processed. You can hook in > any initialisation logic there,

Re: periodic trigger

en.pas...@next-stream.com> wrote: > > Imagine a case where i want to run a computation every X seconds for 1 day > window. I want the calculate average session length for current day every X > seconds. Is there an easy way to achieve that? > > On 21.12.2017 16:06, Piotr Nowojs

Re: periodic trigger

Hi, You defined a tumbling window (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#tumbling-windows ) of 60 seconds, triggered every 10 seconds. This means that each

Re: Metric reporters with non-static ports

I am not sure (and because of holiday season you might not get an answer quickly), however I do not see a way to obtain this port other then by looking into the log files. On the other hand, I have an impression, that intention of this feature was that if you must execute N reporters on one

Re: entrypoint for executing job in task manager

Open method is called just before any elements are processed. You can hook in any initialisation logic there, including initialisation of a static context. However keep in mind, that since this context is static, it will be shared between multiple operators (if you are running parallelism >

Re: Custom Metrics

.addGroup("Users") > .meter(userId, new DropwizardMeterWrapper(new > com.codahale.metrics.Meter())); > Thanks a bunch. > > On Mon, Dec 11, 2017 at 11:12 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, >

Re: Flink flick cancel vs stop

a > savepoint will not wait for the Kafka transactions to bet committed and is > not atomic. > > So it seems like there is a need for an atomic stop or cancel with savepoint > that waits for transactional sinks to commit and then immediately stop any > further message processing. > &

Re: Off heap memory issue

er has 383,120 > instances. I don't know if this could be related. > > I hope that we can test this soon, and will let you know if this fixed the > problem. > > Thanks. > > > On 15 November 2017 at 13:18, Piotr Nowojski <pi...@data-artisans.com > <mailto:p

Re: How to deal with dynamic types

Hi, For truly dynamic class you would need a custom TypeInformation or TypeDeserializationSchema and store the fields on some kind of Map. Maybe something could be done with inheritance if records that always share the same fields could be deserialized to some specific class

Re: Custom Metrics

x and grafana to plot the metrics. > > The option 2 that I had in mind was to collect all metrics and use influx db > sink to report it directly inside the pipeline. But it seems reporting per > node might not be possible. > > > On Mon, Dec 11, 2017 at 3:14 AM, Piotr N

Re: REST api: how to upload jar?

Hi, Have you tried this https://stackoverflow.com/questions/41724269/apache-flink-rest-client-jar-upload-not-working ? Piotrek > On 11 Dec 2017, at 14:22, Edward wrote: > >

Re: Exception when using the time attribute in table API

Hi, NoSuchMethodError probably comes from some mismatched compile/runtime versions of the Flink. Do you have to use 1.4-SNAPSHOT version? It can change on daily basis, so you have to be more careful about what Flink jar's you are using at runtime and what on compile time. If you really need

Re: ayncIO & TM akka response

Hi, Please search the task manager logs for the potential reason of failure/disconnecting around the time when you got this error on the job manager. There should be some clearly visible exception. Thanks, Piotrek > On 9 Dec 2017, at 20:35, Chen Qin wrote: > > Hi there,

Re: Custom Metrics

Hi, I’m not sure if I completely understand your issue. 1. - You don’t have to pass RuntimeContext, you can always pass just the MetricGroup or ask your components/subclasses “what metrics do you want to register” and register them at the top level. - Reporting tens/hundreds/thousands of

Re: How & Where does flink stores data for aggregations.

Hi, Flink will have to maintain state of the defined aggregations per each window and key (the more names you have, the bigger the state). Flink’s state backend will be used for that (for example memory or rocksdb). However in most cases state will be small and not dependent on the length of

Re: Correlation between data streams/operators and threads

I'm not sure how this is different from the current > approach I had provided the code for above, and will it solve this problem of > different data streams not getting distributed across slots? > > Thanks again, > Shailesh > > On Fri, Nov 17, 2017 at 3:01 PM, P

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, If the TM is not responding check the TM logs if there is some long gap in logs. There might be three main reasons for such gaps: 1. Machine is swapping - setup/configure your machine/processes that machine never swap (best to disable swap altogether) 2. Long GC full stops - look how to

Re: Flink memory leak

task managers memory pool with VisualVm and Console, we > attached the screenshot of results. > Could you help us about evaluating the results? > > -Ebru > > >> On 14 Nov 2017, at 19:29, Piotr Nowojski <pi...@data-artisans.com >> <mailto:pi...@data-artisans.com&

Re: Off heap memory issue

Hi, I have been able to observe some off heap memory “issues” by submitting Kafka job provided by Javier Lopez (in different mailing thread). TL;DR; There was no memory leak, just memory pool “Metaspace” and “Compressed Class Space” are growing in size over time and are only rarely garbage

Re: Flink memory leak

gt; > On Tue, Nov 14, 2017 at 4:02 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Ebru, Javier, Flavio: > > I tried to reproduce memory leak by submitting a job, that was generating > classes with random names. And ind

Re: Flink memory leak

encing? If not, I would really need more help/information from you to track this down. Piotrek > On 10 Nov 2017, at 15:12, ÇETİNKAYA EBRU ÇETİNKAYA EBRU > <b20926...@cs.hacettepe.edu.tr> wrote: > > On 2017-11-10 13:14, Piotr Nowojski wrote: >> jobmanager1.log and taskmanager2.

Re: Correlation between data streams/operators and threads

t; attached a snap of my job plan visualization. > > 3. Have attached the logs and exception raised (15min - configured akka > timeout) after submitting the job. > > Thanks, > Shailesh > > > On Tue, Nov 14, 2017 at 2:46 PM, Piotr Nowojski <pi...@data-artisans.com

Re: Correlation between data streams/operators and threads

ave other questions or encounter some issues. > > Thanks, Piotrek > > >> On 13 Nov 2017, at 14:49, Shailesh Jain <shailesh.j...@stellapps.com >> <mailto:shailesh.j...@stellapps.com>> wrote: >> >> Thanks, Piotr. I'll try it out and will get back in case of

Re: Correlation between data streams/operators and threads

lesh > > On Fri, Nov 10, 2017 at 5:52 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > 1. It’s a little bit more complicated then that. Each operator chain/task > will be executed in separate thread (parallelism > Multiplies

Re: Flink memory leak

-pnowojski-taskmanager-9-piotr-mbp.log Description: Binary data > On 10 Nov 2017, at 16:05, ÇETİNKAYA EBRU ÇETİNKAYA EBRU > <b20926...@cs.hacettepe.edu.tr> wrote: > > On 2017-11-10 18:01, ÇETİNKAYA EBRU ÇETİNKAYA EBRU wrote: >> On 2017-11-10 17:50, Piotr Nowojski wrote: &

Re: Flink memory leak

lt;b20926...@cs.hacettepe.edu.tr> wrote: > > On 2017-11-10 13:14, Piotr Nowojski wrote: >> jobmanager1.log and taskmanager2.log are the same. Can you also submit >> files containing std output? >> Piotrek >>> On 10 Nov 2017, at 09:35, ÇETİNKAYA EBRU ÇETİNKAYA EBRU >>&

Re: Flink memory leak

jobmanager1.log and taskmanager2.log are the same. Can you also submit files containing std output? Piotrek > On 10 Nov 2017, at 09:35, ÇETİNKAYA EBRU ÇETİNKAYA EBRU > <b20926...@cs.hacettepe.edu.tr> wrote: > > On 2017-11-10 11:04, Piotr Nowojski wrote: >> Hi, >>

Re: Flink memory leak

BRU > <b20926...@cs.hacettepe.edu.tr> wrote: > > On 2017-11-09 20:08, Piotr Nowojski wrote: >> Hi, >> Could you attach full logs from those task managers? At first glance I >> don’t see a connection between those exceptions and any memory issue >> that you m

Re: Flink memory leak

; > On 2017-11-08 18:30, Piotr Nowojski wrote: >> Btw, Ebru: >> I don’t agree that the main suspect is NetworkBufferPool. On your >> screenshots it’s memory consumption was reasonable and stable: 596MB >> -> 602MB -> 597MB. >> PoolThreadCache memory usage

Re: How to best create a bounded session window ?

n the > leading boundary is forever increasing. We may decide to fire_and_purge. fire > etc but the window remains live. Or did I get that part wrong ? > > > Vishal. > > > > > On Thu, Nov 9, 2017 at 8:24 AM, Piotr Nowojski <pi...@data-artisans.com >

Re: Correlation between data streams/operators and threads

Hi, 1. https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/parallel.html Number of threads executing would be roughly speaking equal to of the number of input data streams multiplied by the

Re: Weird performance on custom Hashjoin w.r.t. parallelism

Hi, Yes as you correctly analysed parallelism 1 was causing problems, because it meant that all of the records must been gathered over the network from all of the task managers. Keep in mind that even if you increase parallelism to “p”, every change in parallelism can slow down your

Re: How to best create a bounded session window ?

It might be more complicated if you want to take into account events coming in out of order. For example you limit length of window to 5 and you get the following events: 1 2 3 4 6 7 8 5 Do you want to emit windows: [1 2 3 4 5] (length limit exceeded) + [6 7 8] ? Or are you fine with

Re: Flink memory leak

ailto:javier.lo...@zalando.de>> wrote: >> >> Yes, I tested with just printing the stream. But it could take a lot of time >> to fail. >> >> On Wednesday, 8 November 2017, Piotr Nowojski <pi...@data-artisans.com >> <mailto:pi...@data-artisans.com>

Re: Flink memory leak

heap size. Piotrek > On 8 Nov 2017, at 15:28, Javier Lopez <javier.lo...@zalando.de> wrote: > > Yes, I tested with just printing the stream. But it could take a lot of time > to fail. > > On Wednesday, 8 November 2017, Piotr Nowojski <pi...@data-artisans.com >

Re: Flink memory leak

need data. With data it will die faster. I tested as well with a > small data set, using the fromElements source, but it will take some time to > die. It's better with some data. > > On 8 November 2017 at 14:54, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@da

Re: Flink memory leak

> "TM_MAX_OFFHEAP_SIZE="2G"; and nothing seems to work. > > Thanks for your help. > > On 8 November 2017 at 13:26, ÇETİNKAYA EBRU ÇETİNKAYA EBRU > <b20926...@cs.hacettepe.edu.tr <mailto:b20926...@cs.hacettepe.edu.tr>> wrote: > On 2017-11-08 15:20, Piotr Nowo

Re: Flink memory leak

Hi Ebru and Javier, Yes, if you could share this example job it would be helpful. Ebru: could you explain in a little more details how does your Job(s) look like? Could you post some code? If you are just using maps and filters there shouldn’t be any network transfers involved, aside from

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

ssor.processInput(StreamInputProcessor.java:213) > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > at org.apache.flink.runtime.taskm

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

Did you try to expose required ports that are listed in the README when starting the containers? https://github.com/apache/flink/tree/master/flink-contrib/docker-flink Ports: • The Web Client is on port 48081

Re: Flink flick cancel vs stop

I would propose implementations of NewSource to be not blocking/asynchronous. For example something like public abstract Future getCurrent(); Which would allow us to perform some certain actions while there are no data available to process (for example flush output buffers). Something like

Re: HBase config settings go missing within Yarn.

ave created a very simple project that reproduces the problem on my setup: > https://github.com/nielsbasjes/FlinkHBaseConnectProblem > <https://github.com/nielsbasjes/FlinkHBaseConnectProblem> > > Niels Basjes > > > On Fri, Oct 20, 2017 at 6:54 PM, Piotr Nowojski <pi.

Re: SLF4j logging system gets clobbered?

Till could you take a look at this? Piotrek > On 18 Oct 2017, at 20:32, Jared Stehler > wrote: > > I’m having an issue where I’ve got logging setup and functioning for my > flink-mesos deployment, and works fine up to a point (the same point every >

Re: flink can't read hdfs namenode logical url

ent.java:619) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) > -- > 发件人：Piotr Nowojski <pi...@data-artisans.com> > 发送时间：2017年10月20日(星期五) 21:39 > 收件人：邓俊华 <deng

Re: HBase config settings go missing within Yarn.

s not have the same settings. > So it seems in the transition into the cluster the application does not copy > everything it has available locally for some reason. > > There is a very high probability I did something wrong, I'm just not seeing > it at this moment. >

Re:

Hi, Only batch API is using managed memory. If you are using streaming API, you can do two things: - estimate max cache size based on for example fraction of max heap size - use WeakReference to implement your cache In batch API, you could estimate max cache size based on: - fraction of

Re: flink can't read hdfs namenode logical url

Hi, Please double check the content of config files in YARN_CONF_DIR and HADOOP_CONF_DIR (the first one has a priority over the latter one) and that they are pointing to correct files. Also check logs (WARN and INFO) for any relevant entries. Piotrek > On 20 Oct 2017, at 06:07, 邓俊华

Re: java.lang.NoSuchMethodError and dependencies problem

gt; > > > >> ---- Оригинално писмо > >> От: Piotr Nowojski pi...@data-artisans.com > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >> До: "r. r." <rob...@abv.bg> > >> Изпратено на: 20.10.2017

Re: Does heap memory gets released after computing windows?

Hi, Memory used by session windows should be released once window is triggered (allowedLateness can prolong window’s life). Unless your code introduces some memory leak (by not releasing references) everything should be garbage collected. Keep in mind that session windows with time gap of 10

Re: HBase config settings go missing within Yarn.

Hi, What do you mean by saying: > When I open the logfiles on the Hadoop cluster I see this: The error doesn’t come from Flink? Where do you execute hbaseConfig.addResource(new Path("file:/etc/hbase/conf/hbase-site.xml")); ? To me it seems like it is a problem with misconfigured HBase and

Re: Flink BucketingSink, subscribe on moving of file into final state

functionality, to handle moments of moving the file into final state. > I thought, that maybe someone has already implemented such thing or knows any > other approaches that will help me to not copy/ paste existing sink impl )) > > Thx ! > > >> On 20 Oct 2017, at 14:37, Piotr

Re: java.lang.NoSuchMethodError and dependencies problem

uld > the class path affect this? > > by shade commons-compress do you mean : > > it doesn't have effect either > > as a last resort i may try to rebuild Flink to use 1.14, but don't want to go > there yet =/ > > > Best regards > > > > > > &

Re: java.lang.NoSuchMethodError and dependencies problem

application pom.xml. I’m not sure if this is solvable in some way, or not. Maybe as a walk around, you could shade commons-compress usages in your pom.xml? Piotr Nowojski > On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > > flink is started with bin

Re: Watermark on connected stream

With aitozi we have a hat trick oO > On 19 Oct 2017, at 17:08, Tzu-Li (Gordon) Tai wrote: > > Ah, sorry for the duplicate answer, I didn’t see Piotr’s reply. Slight delay > on the mail client. > > > On 19 October 2017 at 11:05:01 PM, Tzu-Li (Gordon) Tai

Re: Watermark on connected stream

Hi, As you can see in org.apache.flink.streaming.api.operators.AbstractStreamOperator#processWatermark1 it takes a minimum of both of the inputs. Piotrek > On 19 Oct 2017, at 14:06, Kien Truong wrote: > > Hi, > > If I connect two stream with different watermark,

Re: java.lang.NoSuchMethodError and dependencies problem

Hi, What is the full stack trace of the error? Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster? Piotrek > On 19 Oct 2017, at 13:34, r. r. wrote: > > Hello > I have a job that runs an

Re: Set heap size

Hi, Just log into the machine and check it’s memory consumption using htop or a similar tool under the load. Remember about subtracting Flink’s memory usage and and file system cache. Piotrek > On 19 Oct 2017, at 10:15, AndreaKinn wrote: > > About task manager heap size

Re: SLF4j logging system gets clobbered?

Hi, What versions of Flink/logback are you using? Have you read this: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html#use-logback-when-running-flink-out-of-the-ide--from-a-java-application

Re: Monitoring job w/LocalStreamEnvironment

wrote: > > Hi Piotr, > > Thanks for responding, see below. > >> On Oct 12, 2017, at 7:51 AM, Piotr Nowojski <pi...@data-artisans.com >> <mailto:pi...@data-artisans.com>> wrote: >> >> Hi, >> >> Have you read the following doc? >>

Re: Submitting a job via command line

Good to hear that :) > On 13 Oct 2017, at 14:40, Alexander Smirnov <asmir...@five9.com> wrote: > > Thank you so much, it helped! > > From: Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> > Date: Thursday, October 12, 2017 at

Re: Writing an Integration test for flink-metrics

12.10.2017 14:25, Chesnay Schepler wrote: > You could also write a custom reporter that opens a socket or similar for > communication purposes. > > You can then either query it for the metrics, or even just trigger the > verification in the reporter, > and fail with an error if the

Re: Beam Application run on cluster setup (Kafka+Flink)

Hi, What version of Flink are you using. In earlier 1.3.x releases there were some bugs in Kafka Consumer code. Could you change the log level in Flink to debug? Did you check the Kafka logs for some hint maybe? I guess that metrics like bytes read/input records of this Link application are

Re: Submitting a job via command line

Have you tried this http://mail-archives.apache.org/mod_mbox/flink-user/201705.mbox/%3ccagr9p8bxhljseexwzvxlk+drotyp1yxjy4n4_qgerdzxz8u...@mail.gmail.com%3E

Re: Monitoring job w/LocalStreamEnvironment

Hi, Have you read the following doc? https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html There are some hints regarding testing your application. Especially take a look at the

Re: Beam Application run on cluster setup (Kafka+Flink)

Hi, What do you mean by: > With standalone beam application kafka can receive the message, But in cluster setup it is not working. In your example you are reading the data from Kafka and printing them to console. There doesn’t seems to be anything that writes back to Kafka, so what do you

Re: Writing to an HDFS file from a Flink stream job

nnectors/filesystem_sink.html > > <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/filesystem_sink.html> > > Best, > Aljoscha > >> On 12. Oct 2017, at 14:55, Piotr Nowojski <pi...@data-artisans.com >> <mailto:pi...@data-artisan

Re: Writing to an HDFS file from a Flink stream job

Hi, Maybe this is an access rights issue? Could you try to create and write to same file (same directory) in some other way (manually?), using the same user and the same machine as would Flink job do? Maybe there will be some hint in hdfs logs? Piotrek > On 12 Oct 2017, at 00:19, Isuru

Re: Writing an Integration test for flink-metrics

Hi, Doing as you proposed using JMXReporter (or custom reporter) should work. I think there is no easier way to do this at the moment. Piotrek > On 12 Oct 2017, at 04:58, Colin Williams > wrote: > > I have a RichMapFunction and I'd like to ensure Meter

Re: Kafka 11 connector on Flink 1.3

Hi, Kafka 0.11 connector depends on some API changes for Flink 1.4, so without rebasing the code and solving some small issues it is not possible to use it for 1.3.x. We are about to finalizing the timeframe for 1.4 release, it would be great if you could come back with this question after

Re: Implement bunch of transformations applied to same source stream in Apache Flink in parallel and combine result

Hi, What is the number of events per second that you wish to process? If it’s high enough (~ number of machines * number of cores) you should be just fine, instead of scaling with number of features, scale with number of events. If you have a single data source you still could randomly shuffle

Re: R/W traffic estimation between Flink and Zookeeper

Hi, Are you asking how to measure records/s or is it possible to achieve it? To measure it you can check numRecordsInPerSecond metric. As far if 1000 records/s is possible, it depends on many things like state backend used, state size, complexity of your application, size of the records,

Re: Write each group to its own file

Hi, There is no straightforward way to do that. First of all, the error you are getting is because you are trying to start new application ( env.fromElements(items) ) inside your reduce function. To do what you want, you have to hash partition the products based on category (instead of

Re: Avoid duplicate messages while restarting a job for an application upgrade

while hundred of thousands were duplicated before. > > I planned to reapply/adapt this patch for the 1.3.2 release when we migrate > to it and maybe later to the 1.4 > > I'm open to suggestion or to help/develop this feature upstream if you want. > > > Le lun. 2 oct. 2017 à 1

Re: Sink buffering

What do you mean by "This always depends on checkpointing interval right?”? In TwoPhaseCommitSinkFunction, transactions are being committed on each Flink checkpoint. I guess same applies to GenericWriteAheadSink. The first one just commits/pre-commits the data on checkpoint, second rewrites

Re: Sink buffering

Hi, Do you mean buffer on state and you want to achieve exactly-once HBase sink? If so keep in mind that you will need some kind of transactions support in HBase to make it 100% reliable. Without transactions, buffering messages on state only reduces chance of duplicated records. How much

Re: How flink monitor source stream task(Time Trigger) is running?

You are welcome :) Piotrek > On Oct 2, 2017, at 1:19 PM, yunfan123 wrote: > > Thank you. > "If SourceFunction.run methods returns without an exception Flink assumes > that it has cleanly shutdown and that there were simply no more elements to > collect/create by

Re: Avoid duplicate messages while restarting a job for an application upgrade

ature ? > Do you have any hint or details on this part of that "todo list" ? > > > Le lun. 2 oct. 2017 à 16:50, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> a écrit : > Hi, > > For failures recovery with Kafka 0.9 it

Re: Avoid duplicate messages while restarting a job for an application upgrade

tively developed. Piotr Nowojski > On Oct 2, 2017, at 3:35 PM, Antoine Philippot <antoine.philip...@teads.tv> > wrote: > > Hi, > > I'm working on a flink streaming app with a kafka09 to kafka09 use case which > handles around 100k messages per seconds. > > To

Re: starting query server when running flink embedded

Hi, You can take a look at how is it done in the exercises here . There are example solutions that run on a local environment. I Hope that helps :) Piotrek > On Sep 28, 2017, at 11:22 PM, Henri Heiskanen

Re: state of parallel jobs when one task fails

Hi, Yes, by default Flink will restart all of the tasks. I think that since Flink 1.3, you can configure a FailoverStrategy to change this behavior.

Re: How flink monitor source stream task(Time Trigger) is running?

I am still not sure what do you mean by “thread crash without throw”. If SourceFunction.run methods returns without an exception Flink assumes that it has cleanly shutdown and that there were simply no more elements to collect/create by this task. If it continue working, without throwing an

Re: How flink monitor source stream task(Time Trigger) is running?

Any exception thrown by your SourceFunction will be caught by Flink and that will mark a task (that was executing this SourceFuntion) as failed. If you started some custom threads in your SourceFunction, you have to manually propagate their exceptions to the SourceFunction. Piotrek > On Sep

Re: How flink monitor source stream task(Time Trigger) is running?

We use Akka's DeathWatch mechanism to detect dead components. TaskManager failure shouldn’t prevent recovering from state (as long as there are enough task slots). I’m not sure if I understand what you mean by "source stream thread" crash. If is was some error during performing a checkpoint so

Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Hi, BucketingSink doesn’t support the feature that you are requesting, you can not specify a dynamically generated prefix/suffix. Piotrek > On Aug 31, 2017, at 7:12 PM, Raja.Aravapalli > wrote: > > > Hi, > > I have a flink application that is streaming data

<    1   2   3   4   5   6   7   >