Re: Multiple stream operator watermark handling

arsome.lucid...@gmail.com> wrote: > > On Thu, May 24, 2018 at 9:20 AM, Elias Levy <fearsome.lucid...@gmail.com > <mailto:fearsome.lucid...@gmail.com>> wrote: > On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com&

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

mer proposal can work. > > I was originally register timer like this. ProcessingTimeCallback interface > doesn't supply the Collector parameter > > ((StreamingRuntimeContext) getRuntimeContext()) > .getProcessingTimeService() > .registerTimer(..., this); > >

Re: Conceptual question

gration state like the manner showed in `WindowOperator`? > > Best Regards, > Tony Wei > > 2018-06-07 20:28 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>: > What function are you implementing and how are you using it? > > Usually it’s enou

Re: Conceptual question

e migration that you > mentioned? > And are there something concerned and difficulties that will leads to > restored state failed or other problems? Thank you! > > Best Regards, > Tony Wei > > 2018-06-07 16:10 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>

Re: Conceptual question

gt; > Best Regards, > Tony Wei > > 2018-06-07 21:43 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>: > Hi, > > Oh, I see now. Yes indeed getKeyedStateBackened() is not exposed to the > function and you can not migrate your state that way. > >

Re: [flink-connector-filesystem] OutOfMemory in checkpointless environment

Hi, BucketingSink is designed to provide exactly-once writes to file system, which is inherently tied to checkpointing. As you just saw, without checkpointing, BucketingSink is never notified that it can commit pending files. If you do not want to use checkpointing for some reasons, you could

Re: [BucketingSink] notify on moving into pending/ final state

.} > public BucketingSink > registerOnPendingStateChangeCallback(FileStateChangeCallback... callbacks) > {...} > > I’m ready to discuss the best ways, how such hooks could be implemented in > the core impl or any other improvements, that will help us to add such > functionality into ou

Re: Flink kafka consumer stopped committing offsets

Hi, What’s your KafkaConsumer configuration? Especially values for: - is checkpointing enabled? - enable.auto.commit (or auto.commit.enable for Kafka 0.8) / auto.commit.interval.ms - did you set setCommitOffsetsOnCheckpoints() ? Please also refer to

Re: [BucketingSink] notify on moving into pending/ final state

Hi, I see that could be a useful feature. What exactly now is preventing you from inheriting from BucketingSink? Maybe it would be just enough to make the BucketingSink easier extendable. One thing now that could collide with such feature is that Kostas is now working on larger BucketingSink

Re: Heap Problem with Checkpoints

Hi, What kind of messages are those “logs about S3 operations”? Did you try to google search them? Maybe it’s a known S3 issue? Another approach is please use some heap space analyser from which you can backtrack classes that are referencing those “memory leaks” and again try to google any

Re: Take elements from window

Hi, Do I understand you correctly, that you just want to have a three different sliding windows (for 3 rules) with duration of 10, 20 and 30 minutes? If so, I haven’t tested it but I would guess that there are at least two solutions for the problem: 1. just create three different sliding

Re: Kafka to Flink to Hive - Writes failing

Yes, BucketingSink is a better option. You can start from looking at the BucketingSink java docs. Please also take a look on this: https://stackoverflow.com/questions/47669729/how-to-write-to-orc-files-using-bucketingsink-in-apache-flink

Re: Akka version conflict running on Flink cluster

Hi, Please take a look on this thread first: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Akka-Http-used-in-custom-RichSourceFunction-td20314.html

Re: Flink kafka consumer stopped committing offsets

atic way to obfuscate the logs so that I'm sure > that there isn't anything sensitive left. Any way, there isn't anything else > to share really. I wrote: "As you can see, it didn't log anything until > ~2018-06-07 22:08. Also that's where the log ends". > > Thanks once

Re: Odd job failure

t; On Mon, May 28, 2018 at 1:48 AM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Most likely suspect is the standard java problem of some dependency > convergence issue. Please check if you are not pulling in multiple Kafka > versions into your class path. Especially

Re: Question about JVM exit caused by timeout exception with the asynchronous IO of flink 1.4.2

Hi, You can increase a timeout, that’s one way to tackle it. In Flink 1.6.0 there will be possibility to override default Flink’s behaviour regarding handling timeouts: https://issues.apache.org/jira/browse/FLINK-7789 to handle them, instead

Re: Extending stream events with a an aggregate value

Hi, Ńo worries :) You probably need to write your own process function to do exactly that, maybe something like this: DataStream> test; DataStream> max = test.keyBy(0) .process(new KeyedProcessFunction, Tuple3>() { public ValueState max; @Override public void

Re: Conceptual question

Hi, General solution for state/schema migration is under development and it might be released with Flink 1.6.0. Before that, you need to manually handle the state migration in your operator’s open method. Lets assume that your OperatorV1 has a state field “stateV1”. Your OperatorV2 defines

Re: Odd job failure

/flink-docs-master/ops/config.html#task-checkpoint-alignment-max-size > > <https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#task-checkpoint-alignment-max-size> > > On Wed, May 2, 2018 at 1:37 PM, Piotr Nowojski <pi...@data-artisans.com > <ma

Re: Odd job failure

Hi, It might be some Kafka issue. From what you described your reasoning seems sound. For some reason TM3 fails and is unable to restart and process any data, thus forcing spilling on checkpoint barriers on TM1 and TM2. I don’t know the reason behind java.lang.NoClassDefFoundError:

Re: Insert data into Cassandra without Flink Cassandra connection

Hi, The only way that I can think of is if you keep your flatMap operator with parallelism 1, but that might defeat the purpose. Otherwise there is no way to open one single connection and share it across multiple TaskManagers (which can be running on different physical machines). Please

Re: Fat jar fails deployment (streaming job too large)

Short answer: could be that your job is simply too big to be serialised, distributed and deserialised in the given time and you would have to increase timeouts even more. Long answer: Do you have the same problem when you try to submit smaller job? Does your cluster work for simpler jobs?

Re: Apache Flink - Flink Forward SF 2018 - Scaling stream data pipelines (source code)

Hi, Till, do have this code somewhere? M Singh: Till is out of the office and will be back on next week, so he will probably not be able to respond for couple of days. Piotrek > On 30 Apr 2018, at 13:51, M Singh wrote: > > Hi: > > I was looking at the flink-forward sf

Re: ConnectedIterativeStreams and processing state 1.4.2

Hi, Why can not you use simple CoProcessFunction and handle cache updates within it’s processElement1 or processElement2 method? Piotrek > On 1 May 2018, at 10:20, Lasse Nedergaard wrote: > > Hi. > > I have a case where I have a input stream that I want to enrich

Re: KafkaProducer with generic (Avro) serialization schema

Hi, My Scala knowledge is very limited (and my Scala's serialization knowledge is non existent), but one way or another you have to make your SerializationSchema serialisable. If indeed this is the problem, maybe a better place to ask this question is on Stack Overflow or some scala specific

Re: intentional back-pressure (or a poor man's side-input)

Maybe it could work with Flink’s 1.5 credit base flow control. But you would need a way to express state “block one input side of the CoProcessFunction”, pass this information up to the input gate and handle it probably similar to how `org.apache.flink.streaming.runtime.io.CachedBufferBlocker`

Re: java.lang.NoSuchMethodError and dependencies problem

uld > the class path affect this? > > by shade commons-compress do you mean : > > it doesn't have effect either > > as a last resort i may try to rebuild Flink to use 1.14, but don't want to go > there yet =/ > > > Best regards > > > > > > &

Re: Watermark on connected stream

Hi, As you can see in org.apache.flink.streaming.api.operators.AbstractStreamOperator#processWatermark1 it takes a minimum of both of the inputs. Piotrek > On 19 Oct 2017, at 14:06, Kien Truong wrote: > > Hi, > > If I connect two stream with different watermark,

Re: java.lang.NoSuchMethodError and dependencies problem

Hi, What is the full stack trace of the error? Are you sure that there is no commons-compresss somewhere in the classpath (like in the lib directory)? How are you running your Flink cluster? Piotrek > On 19 Oct 2017, at 13:34, r. r. wrote: > > Hello > I have a job that runs an

Re: Watermark on connected stream

With aitozi we have a hat trick oO > On 19 Oct 2017, at 17:08, Tzu-Li (Gordon) Tai wrote: > > Ah, sorry for the duplicate answer, I didn’t see Piotr’s reply. Slight delay > on the mail client. > > > On 19 October 2017 at 11:05:01 PM, Tzu-Li (Gordon) Tai

Re: java.lang.NoSuchMethodError and dependencies problem

application pom.xml. I’m not sure if this is solvable in some way, or not. Maybe as a walk around, you could shade commons-compress usages in your pom.xml? Piotr Nowojski > On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > > flink is started with bin

Re: java.lang.NoSuchMethodError and dependencies problem

gt; > > > >> ---- Оригинално писмо > >> От: Piotr Nowojski pi...@data-artisans.com > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >> До: "r. r." <rob...@abv.bg> > >> Изпратено на: 20.10.2017

Re: flink can't read hdfs namenode logical url

Hi, Please double check the content of config files in YARN_CONF_DIR and HADOOP_CONF_DIR (the first one has a priority over the latter one) and that they are pointing to correct files. Also check logs (WARN and INFO) for any relevant entries. Piotrek > On 20 Oct 2017, at 06:07, 邓俊华

Re:

Hi, Only batch API is using managed memory. If you are using streaming API, you can do two things: - estimate max cache size based on for example fraction of max heap size - use WeakReference to implement your cache In batch API, you could estimate max cache size based on: - fraction of

Re: HBase config settings go missing within Yarn.

s not have the same settings. > So it seems in the transition into the cluster the application does not copy > everything it has available locally for some reason. > > There is a very high probability I did something wrong, I'm just not seeing > it at this moment. >

Re: flink can't read hdfs namenode logical url

ent.java:619) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) > -- > 发件人：Piotr Nowojski <pi...@data-artisans.com> > 发送时间：2017年10月20日(星期五) 21:39 > 收件人：邓俊华 <deng

Re: SLF4j logging system gets clobbered?

Till could you take a look at this? Piotrek > On 18 Oct 2017, at 20:32, Jared Stehler > wrote: > > I’m having an issue where I’ve got logging setup and functioning for my > flink-mesos deployment, and works fine up to a point (the same point every >

Re: HBase config settings go missing within Yarn.

ave created a very simple project that reproduces the problem on my setup: > https://github.com/nielsbasjes/FlinkHBaseConnectProblem > <https://github.com/nielsbasjes/FlinkHBaseConnectProblem> > > Niels Basjes > > > On Fri, Oct 20, 2017 at 6:54 PM, Piotr Nowojski <pi.

Re: Datastream broadcast with KeyBy

Hi, Could you elaborate what is the problem that you are having? What is the exception(s) that you are getting? I have tested such simple example and it’s seems to be working as expected: DataStreamSource input = env.fromElements(1, 2, 3, 4, 5, 1, 2, 3); DataStreamSource confStream =

Re: Stream job failed after increasing number retained checkpoints

ns.com>> wrote: > Hi, > > there is no known limitation in the strict sense, but you might run out of > dfs space or job manager memory if you keep around a huge number checkpoints. > I wonder what reason you might have that you ever want such a huge number of > retaine

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

Hi, > > I attached log file, > > Thanks > > On Mon, Jan 15, 2018 at 3:36 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Could you post full job manager and task manager logs from startup until the &g

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

ce.InstanceManager - Registered > TaskManager at ??? (akka://flink/deadLetters) as > 626846ae27a833cb094eeeb047a6a72c. Current number of registered hosts is 2. > Current number of alive task slots is 40. > > > On Wed, Jan 10, 2018 at 11:36 AM, Piotr Nowojski <pi...@data-artisans.com >

Re: Send ACK when all records of file are processed

.addSink(// DB) > > instead of add sink, should it be a simple map operator which writes to DB > so that we can have a next ack operator which will generate the response. > > Also, how do I get/access the Watermark value in the ack operator ? It will > be a simple map

Re: Data exchange between tasks (operators/sources) at streaming api runtime

Hi, As far as I know there is currently no simple way to do this: Join stream with static data in https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API and

Re: Avoiding deadlock with iterations

Hi, This is a known problem and I don’t think there is an easy solution to this. Please refer to the: http://mail-archives.apache.org/mod_mbox/flink-user/201704.mbox/%3c5486a7fd-41c3-4131-5100-272825088...@gaborhermann.com%3E

Re: Send ACK when all records of file are processed

Hi, As you figured out, some dummy EOF record is one solution, however you might try to achieve it also by wrapping an existing CSV function. Your wrapper could emit this dummy EOF record. Another (probably better) idea is to use Watermark(Long.MAX_VALUE) for the EOF marker. Stream source

Re: Data exchange between tasks (operators/sources) at streaming api runtime

> Thanks for the pointers. > Ishwara Varnasi > > Sent from my iPhone > > On Jan 25, 2018, at 4:03 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > >> Hi, >> >> As far as I know there is currently

Re: Stream job failed after increasing number retained checkpoints

Hi, Increasing akka’s timeouts is rarely a solution for any problems - it either do not help, or just mask the issue making it less visible. But yes, it is possible to bump the limits:

Re: Custom Partitioning for Keyed Streams

Hi, I don’t think it is possible to enforce scheduling of two keys to different nodes, since all of that is based on hashes. For some cases, doing the pre-aggregation step (initial aggregation done before keyBy, which is followed by final aggregation after the keyBy) can be the solution for

Re: Send ACK when all records of file are processed

the > LONG.MAX_VALUE, it will get complicated. > > > > Regards, > Vinay Patil > > On Tue, Jan 30, 2018 at 1:57 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Thanks for the clarification :) > > Since you hav

Re: Latest version of Kafka

Hi, Flink as for now provides only a connector for Kafka 0.11, which is using KafkaClient in 0.11.x version. However you should be able to use it for reading to/writing from Kafka 1.0 - Kafka claims (and as far as I know it’s true) that Kafka 1.0 is backward compatible with 0.11. Piotrek >

Re: Why does jobmanager running needs slot ?

ow. Is it a new design at FLIP6 ? > > Rice. > > > > > > At 2018-02-05 17:49:05, "Piotr Nowojski" <pi...@data-artisans.com> wrote: > I might be wrong but I think it is other way around and the naming of this > method is correct - it does exactly

Re: Why does jobmanager running needs slot ?

implemented this before. > > Rice. > > > > > > At 2018-02-05 17:56:49, "Piotr Nowojski" <pi...@data-artisans.com> wrote: > It seems so - but I’m saying this only basing on a annotations when this > method was added (in the last couple of months). I’m no

Re: Why does jobmanager running needs slot ?

e better to rename it to > requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I > just feel like offerSlotToJobManager sounds strange.. What do you think of > this ? > > Rice. > > > > > > At 2018-02-05 17:30:3

Re: Flink not writing last few elements to disk

Hi, FileProcessMode.PROCESS_CONTINUOUSLY processes the file continuously - the stream will not end. Simple `writeAsCsv(…)` on the other hand only flushes the output file on a stream end (see `OutputFormatSinkFunction`). You can either use `PROCESS_ONCE` mode or use more advanced data sink: -

Re: Reduce parallelism without network transfer.

Hi, It should work like this out of the box if you use rescale method: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html#physical-partitioning

Re: Global window keyBy

Hi, FIRE_AND_PURGE triggers `org.apache.flink.api.common.state.State#clear()` call and it "Removes the value mapped under the current key.”. So other keys should remain unmodified. I hope this solves your problem/question? Piotrek > On 4 Feb 2018, at 15:39, miki haiat

Re: Getting Key from keyBy() in ProcessFunction

I think now it’s not easily possible, however it might be a valid suggestion to add `OnTimerContext#getCurrentKey()` method. Besides using ValueState as you discussed before, as a some kind of a walk around you could copy and modify KeyedProcessOperator to suits your needs, but this would be

Re: Why does jobmanager running needs slot ?

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager. In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a

Re: Python and Scala

Hi, 1. Flink’s Python Batch API is not complete and it’s not on pair with Scala. 2. As for know there is no Python API for Flink Streaming, however there is some ongoing work with that: https://issues.apache.org/jira/browse/FLINK-5886 3. CEP

Re: Ceph configuration for checkpoints?

Hi, Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be: > a drop-in replacement for the Hadoop File System (HDFS) So I would first try to configure ceph according to their documentation

Re: Python and Scala

Hi Scala REPL uses the same code as compiled library so they should work the same. Piotrek > On 13 Feb 2018, at 18:32, Esa Heikkinen <heikk...@student.tut.fi> wrote: > > Hi > > And what about the differences between Scala REPL and Scala (compiled) ? > Esa >

Re: Python and Scala

don’t get it > to work yet. > > Esa > > From: Piotr Nowojski [mailto:pi...@data-artisans.com] > Sent: Wednesday, February 14, 2018 10:55 AM > To: Esa Heikkinen <esa.heikki...@student.tut.fi> > Cc: Esa Heikkinen <esa.heikki...@student.tut.fi>; user@flink.apache.o

Re: Reduce parallelism without network transfer.

ducing parallelism is rebalance, and > not rescale ? > > Regards, > Kien > > Sent from TypeApp <http://www.typeapp.com/r?b=11979> > On Feb 5, 2018, at 15:28, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > &

Re: Rebalance to subtasks in same TaskManager instance

Hi, Unfortunately I don’t think it’s currently possible in the Flink. Please feel free to submit a feature request for it on our JIRA https://issues.apache.org/jira/projects/FLINK/summary Have you tried out the setup using rebalance? In

Re: periodic trigger

en.pas...@next-stream.com> wrote: > > Imagine a case where i want to run a computation every X seconds for 1 day > window. I want the calculate average session length for current day every X > seconds. Is there an easy way to achieve that? > > On 21.12.2017 16:06, Piotr Nowojs

Re: entrypoint for executing job in task manager

; before any operator opening. > > On Thu, Dec 21, 2017 at 12:27 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Open method is called just before any elements are processed. You can hook in > any initialisation logic there,

Re: periodic trigger

Hi, You defined a tumbling window (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#tumbling-windows ) of 60 seconds, triggered every 10 seconds. This means that each

Re: periodic trigger

nt to > discard events from different users received later then i might receive more > than one row for a single user based on the number of windows created by the > events of this user. That will make the the average computations wrong. > > On 22.12.2017 12:10, Piotr Nowojski wrote: >

Re: Heap Problem with Checkpoints

? I > tried also using a bigger instance, but this never stabilizes, it just keeps > increasing (gave it already 10GB+ Heap) ... > > Cheers > > -- > > Fabian Wollert > Zalando SE > > E-Mail: fabian.woll...@zalando.de <mailto:fabian.woll...@zalando.de> > >

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, Are those paths: file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing) file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar' (missing) accessible from the inside of your container? bin/flink run --help (…) -C,--classpath Adds a URL to each

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

b can not run in cluster on yarn, yt just > can not submit. > > I am trying to change the "-C > file:/home/work/xxx/lib/commons-math3-3.5.jar" to "-C > hdfs://namenode1/home/work/xxx/lib/commons-math3-3.5.jar", but Clifrontend > error was caught. > I

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

t; > It seems I have to find a way to shorten the time my users cost. > > Yours Sincerely > > Joshua > > On Fri, Aug 3, 2018 at 9:08 PM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi, > > -yt,--yarnship Ship files in the spe

Re: Heap Problem with Checkpoints

Hi, Thanks for getting back with more information. Apparently this is a known bug of JDK since 2003 and is still not resolved: https://bugs.java.com/view_bug.do?bug_id=4872014 https://bugs.java.com/view_bug.do?bug_id=6664633

Re: Scala 2.12 Support

Hi, Scala 2.12 support is high on our priority list and we hope to have it included for the 1.7 release (as you can see in the ticket itself), which should happen later this year. Piotrek > On 15 Aug 2018, at 17:59, Aaron Levin wrote: > > Hello! > > I'm wondering if there is anywhere I can

Re: How to compare two window ?

Hi, Could you rephrase your question? Maybe by posting some code examples? Piotrek > On 16 Aug 2018, at 08:26, 苗元君 wrote: > > Hi, Flink guys, > U really to a quick release, it's fantastic ! > > I'v got a situation , > window 1 is time driven, slice is 1min, trigger is 1 count > window 2

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

Hi, You made a small mistake when restoring from state using test harness, that I myself have also done in the past. Problem is with an ordering of those calls: result.open(); if (savedState != null) { result.initializeState(savedState); } Open is supposed

Re: How to submit flink job on yarn by java code

Hi, Is this path accessible on the container? If not, use some distributed file system, nfs or -yt —yarnship option of the cli. Please also take a look at https://lists.apache.org/thread.html/%3CCAF=1nJ8GONoqux7czxpUxAf7L3p=-E_ePSTHk0uWa=GRyG=2...@mail.gmail.com%3E

Re: Standalone cluster instability

y. > > Are there any heuristics or best practices which you (or anyone in the > community) recommend to benchmark memory requirements of a particular flink > job? > > Thanks, > Shailesh > > > On Tue, Aug 14, 2018 at 6:08 PM, Piotr Nowojski <mailto:pi...@data-

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

g I should have noticed. > > — Ken > > >> On Aug 16, 2018, at 4:37 AM, Piotr Nowojski > <mailto:pi...@data-artisans.com>> wrote: >> >> Hi, >> >> You made a small mistake when restoring from state using test harness, that >> I myself

Re: Standalone cluster instability

nd anything special in the logs before the failure. > Here are the logs, please take a look: > > https://drive.google.com/drive/folders/1zlUDMpbO9xZjjJzf28lUX-bkn_x7QV59?usp=sharing > > <https://drive.google.com/drive/folders/1zlUDMpbO9xZjjJzf28lUX-bkn_x7QV59?usp=sharing> > > T

Re: Standalone cluster instability

fka source/sink, ~8 Window operators, and the rest CEP > operators). > > Thanks, > Shailesh > > On Fri, Aug 10, 2018 at 4:48 PM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Please post full TaskManager logs, including stderr and stdout. (Have

Re: Flink job hangs/deadlocks (possibly related to out of memory)

ks, > > Gerard > > On Mon, Jul 16, 2018 at 4:12 PM Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi Gerard, > > I second to what Zhijiang wrote. Please check GC pauses, either via GC > logging, 3rd party tool like jconsole (or some

Re: Flink job hangs/deadlocks (possibly related to out of memory)

Hi Gerard, I second to what Zhijiang wrote. Please check GC pauses, either via GC logging, 3rd party tool like jconsole (or some memory profiler) or via enabling resource logging in Flink. After confirming that this is not the issue next time this happens, instead of cancelling the job,

Re: Extremely large job serialization produced by union operator

Hi, Could you provide more details about your queries and setup? Logs could be helpful as well. Piotrek > On 9 Mar 2018, at 11:00, 杨力 wrote: > > I wrote a flink-sql app with following topography. > > KafkaJsonTableSource -> SQL -> toAppendStream -> Map ->

Re: Slow Flink program

Hi, First of all learn about what’s going with your job: check the status of the machines, cpu/network usage on the cluster. If CPU is not ~100%, analyse what is preventing the machines to work faster (network bottleneck, locking, blocking operations etc). If CPU is ~100%, profile the

Re: Hi Flink Team

Hi, timeWindowAll is a non parallel operation, since it gathers all of the elements and process them together:

Re: Too many open files on Bucketing sink

Hi, There is an open similar issue: https://issues.apache.org/jira/browse/FLINK-8707 It’s still under investigation and it would be helpful if you could follow up the discussion there, run same diagnostics commands as Alexander Gardner did

Re: Emulate Tumbling window in Event Time Space

Hi, As Xingcan responded, you could use already built in operator for that. If you really want to implement something on your own (need custom feature? For fun?), you would have to implement some variation of a InternalTimerService from Flink (you can browse the code for an inspiration). On

Re: "Close()" aborts last transaction in TwoPhaseCommitSinkFunction

Hi, Short answer is: no, at the moment clean shutdown is not implemented for the streaming, but it’s on our to do list for the future. Hacky answer: you could implement some custom code, that would wait for at least one completed checkpoint after the last input data. But that would require

Re: Restart strategy defined in flink-conf.yaml is ignored

ckpointingMode.EXACTLY_ONCE); > > DataStream stream = env.fromCollection(Arrays.asList("test")); > > stream.map(new MapFunction<String, String>(){ > @Override > public String map(String obj) { > throw new Null

Re: Checkpoints very slow with high backpressure

-the-network-buffers <https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html#configuring-the-network-buffers> On the other hand, why does it prevents you from using a checkpointing at all? Piotr Nowojski > On 5 Apr 2018, at 06:10, Edward <egb...@hotmail.com>

Re: Restart strategy defined in flink-conf.yaml is ignored

Hi, Can you provide more details, like post your configuration/log files/screen shots from web UI and Flink version being used? Piotrek > On 5 Apr 2018, at 06:07, Alexander Smirnov > wrote: > > Hello, > > I've defined restart strategy in flink-conf.yaml as

Re: Checkpoints very slow with high backpressure

Thanks for the explanation. I hope that either 1.5 will solve your issue (please let us know if it doesn’t!) or if you can’t wait, that decreasing memory buffers can mitigate the problem. Piotrek > On 5 Apr 2018, at 08:13, Edward wrote: > > Thanks for the update Piotr. >

Re: Restart strategy defined in flink-conf.yaml is ignored

> > > Alex > > > On Thu, Apr 5, 2018 at 11:28 PM Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Thanks for the details! I can confirm this behaviour. flink-conf.yaml > restart-strategy value is being co

Re: Kafka ProducerFencedException after checkpointing

Hi, What’s your Kafka’s transaction timeout setting? Please both check Kafka producer configuration (transaction.timeout.ms property) and Kafka broker configuration. The most likely cause of such error message is when Kafka's timeout is smaller then Flink’s checkpoint interval and transactions

Re: Checkpoint is not triggering as per configuration

uhamamd Abrar Akber > MS110400027 > > On Mon, Feb 5, 2018 at 5:33 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Did you check task manager and job manager logs for any problems? > > Piotrek > > &

Re: Kafka ProducerFencedException after checkpointing

t; Anyway thank you very much for the detailed explanation! > > Best, > > Dongwon > > > > On Tue, Mar 20, 2018 at 8:10 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Please increase transaction.timeo

Re: Confluent Schema Registry DeserializationSchema

Hi, It looks like to me that kafka.utils.VerifiableProperties comes from org.apache.kafka:kafka package - please check and solve (if possible) dependency conflicts in your pom.xml regarding this package. Probably there is some version collision. Piotrek > On 21 Mar 2018, at 16:40, dim5b

Re: Error running on Hadoop 2.7

Hi, Have you replaced all of your old Flink binaries with freshly downloaded Hadoop 2.7 versions? Are you sure that something hasn't mix in the process? Does some simple word count example works on the cluster after the upgrade? Piotrek > On 21 Mar

<    1   2   3   4   5   6   7   >