Re: FlinkKafkaProducer API

On Thu, Jun 9, 2016 at 5:16 AM, Fabian Hueske wrote: > thanks for your feedback. I think those are good observations and > suggestions to improve the Kafka producers. > The best place to discuss such improvements is the dev mailing list. > > Would like to repost your mail

Re: Kafka producer sink message loss?

On Tue, Jun 7, 2016 at 4:52 AM, Stephan Ewen wrote: > The concern you raised about the sink being synchronous is exactly what my > last suggestion should address: > > The internal state backends can return a handle that can do the sync in a > background thread. The sink would

Arrays values in keyBy

I would be useful if the documentation warned what type of equality it expected of values used as keys in keyBy. I just got bit in the ass by converting a field from a string to a byte array. All of the sudden the windows were no longer aggregating. So it seems Flink is not doing a deep compare

Kafka producer sink message loss?

I am correct in assuming that the Kafka producer sink can lose message? I don't expect exactly-once semantics using Kafka as a sink given Kafka publishing guarantees, but I do expect at least once. I gather from reading the source that the producer is publishing messages asynchronously, as

Re: Kafka producer sink message loss?

On Sun, Jun 5, 2016 at 3:16 PM, Stephan Ewen wrote: > You raised a good point. Fortunately, there should be a simply way to fix > this. > > The Kafka Sunk Function should implement the "Checkpointed" interface. It > will get a call to the "snapshotState()" method whenever a

Re: Event processing time with lateness

On Fri, Jun 3, 2016 at 6:47 AM, Kostas Kloudas wrote: > To see a relatively more complex example of such a trigger and how to > implement it, > you can have a look at this implementation: >

FlinkKafkaProducer API

The FlinkKafkaProducer API seems more difficult to use than it should be. The API requires you pass it a SerializationSchema or a KeyedSerializationSchema, but the Kafka producer already has a serialization API. Requiring a serializer in the Flink API precludes the use of the Kafka serializers.

WindowedStream aggregation methods pre-aggregate?

Can someone confirm whether the org.apache.flink.streaming.api.scala.WindowedStream methods other than "apply" (e.g. "sum") perform pre-aggregation? The API docs are silent on this.

WindowedStream operation questions

An observation and a couple of question from a novice. The observation: The Flink web site makes available ScalaDocs for org.apache.flink.api.scala but not for org.apache.flink.streaming.api.scala. Now for the questions: Why can't you use map to transform a data stream, say convert all the

Access an event's TimeWindow?

Is there an API to access an event's time window? When you are computing aggregates over a time window, you usually want to output the window along with the aggregate. I could compute the Window on my own, but this seems like an API that should exist.

WindowedStream sum behavior

I am wondering about the expected behavior of the sum method. Obviously it sums a specific field in a tuple or POJO. But what should one expect in other fields? Does sum keep the first field, last field or there aren't any guarantees?

Does Kafka connector leverage Kafka message keys?

I am wondering if the Kafka connectors leverage Kafka message keys at all? Looking through the docs my impression is that it does not. E.g. if I use the connector to consume from a partitioned Kafka topic, what I will get back is a DataStream, rather than a KeyedStream. And if I want access to

Re: How to perform this join operation?

. In my data set keys come and go, and many will never be observed again. That will lead to continuous state growth over time. On Mon, May 2, 2016 at 6:06 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > Thanks for the suggestion. I ended up implementing it a different way. > > [

TimeWindow overload?

Looking over the code, I see that Flink creates a TimeWindow object each time the WindowAssigner is created. I have not yet tested this, but I am wondering if this can become problematic if you have a very long sliding window with a small slide, such as a 24 hour window with a 1 minute slide. It

Re: How to perform this join operation?

Till, Thanks again for putting this together. It is certainly along the lines of what I want to accomplish, but I see some problem with it. In your code you use a ValueStore to store the priority queue. If you are expecting to store a lot of values in the queue, then you are likely to be using

Re: How to perform this join operation?

Thanks for the suggestion. I ended up implementing it a different way. What is needed is a mechanism to give each stream a different window assigner, and then let Flink perform the join normally given the assigned windows. Specifically, for my use case what I need is a sliding window for one

Re: How to perform this join operation?

tState. As I > understand the drawback is that the list state is not maintained in the > managed memory. > I'm interested to hear about the right way to implement this. > > On Wed, Apr 13, 2016 at 3:53 PM, Elias Levy <fearsome.lucid...@gmail.com> > wrote: > >> I am wondering h

Re: Does Kafka connector leverage Kafka message keys?

On Wed, Apr 13, 2016 at 2:10 AM, Stephan Ewen wrote: > If you want to use Flink's internal key/value state, however, you need to > let Flink re-partition the data by using "keyBy()". That is because Flink's > internal sharding of state (including the re-sharding to adjust

Does Flink DataStreams using combiners?

I am wondering if Flink makes use of combiners to pre-reduce a keyed and windowed stream before shuffling the data among workers. I.e. will it use a combiner in something like: stream.flatMap {...} .assignTimestampsAndWatermarks(...) .keyBy(...) .timeWindow(...)

Re-keying / sub-keying a stream without repartitioning

This is something that has come up before on the list, but in a different context. I have a need to rekey a stream but would prefer the stream to not be repartitioned. There is no gain to repartitioning, as the new partition key is a composite of the stream key, going from a key of A to a key of

CEP join across events

There doesn't appear to be a way to join events across conditions using the CEP library. Consider events of the form (type, value_a, value_b) on a stream keyed by the value_a field. Under 1.2 you can create a pattern that for a given value_a, as specified by the stream key, there is a match if

Re: Re-keying / sub-keying a stream without repartitioning

On Wed, Apr 26, 2017 at 5:11 AM, Aljoscha Krettek wrote: > I did spend some time thinking about this and we had the idea for a while > now to add an operation like “keyByWithoutPartitioning()” (name not final > ;-) that would allow the user to tell the system that we don’t

Re: CEP join across events

ched > in "foo" pattern in that particular branch. > I mean that for a sequence like (type =1, value_b = X); (type=1, > value_b=Y); (type=2, value_b=X) both events of type = 1 create a seperate > pattern branch and the event with type = 2 will be checked for a matc

Re: Re-keying / sub-keying a stream without repartitioning

Anyone? On Fri, Apr 21, 2017 at 10:15 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > This is something that has come up before on the list, but in a different > context. I have a need to rekey a stream but would prefer the stream to > not be repartitioned. T

Re: CEP join across events

It would be useful if there were a cleaner syntax for specifying relationships between matched events, as in an SQL join, particularly for conditions with a quantifier of one. At the moment you have to do something like Pattern. .begin[Foo]("first") .where( first => first.baz

Re: CEP memory requirements

Looking at the code I gather that 1.2 does not clear the per key NFA state even if there is no state to keep, whereas this appears fixed in the master branch. Yes? On Thu, May 4, 2017 at 11:25 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I am observing odd memory behavior wit

Re: RocksDB error with flink 1.2.0

Any reason they can't share a single RocksDB state backend instance? On Fri, Apr 28, 2017 at 8:44 AM, Aljoscha Krettek wrote: > The problem here is that this will try to open 300 RocksDB instances on > each of the TMs (depending on how the parallelism is spread between the

CEP memory requirements

I am observing odd memory behavior with the CEP library and I am wondering if it is expected. If I write a simple local streaming Flink job that reads from a 65MB compressed file of JSON objects, one per line, parses the JSON, performs a filter operation, and then a keyBy, heap usage is stable,

Flink flick cancel vs stop

I was wondering about the status of the flink stop command. At first blush it would seem as the preferable way to shutdown a Flink job, but it depends on StoppableFunction being implemented by sources and I notice that the Kafka source does not seem to implement it. In addition, the command does

Re: Flink flick cancel vs stop

On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright wrote: > Aljoscha, would it be correct to characterize your idea as a 'pull' source > rather than the current 'push'? It would be interesting to look at the > existing connectors to see how hard it would be to reverse their >

History Server

I am curious, why is the History Server a separate process and Web UI instead of being part of the Web Dashboard within the Job Manager?

high-availability.jobmanager.port vs jobmanager.rpc.port

I am wondering why HA mode there is a need for a separate config parameter to set the JM RPC port (high-availability.jobmanager.port) and why this parameter accepts a range, unlike jobmanager.rpc.port.

Re: Flink flick cancel vs stop

Anyone? On Mon, Sep 11, 2017 at 6:17 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I was wondering about the status of the flink stop command. At first > blush it would seem as the preferable way to shutdown a Flink job, but it > depends on StoppableFunction being impleme

Re: Assigning operators to slots

The execution within the IDE is most likely not loading the flink-conf.yaml file to read the configuration. When run from the IDE you get a LocalStreamEnvironment, which starts a LocalFlinkMiniCluster. LocalStreamEnvironment is created by StreamExecutionEnvironment.createLocalEnvironment without

Re: Noisy org.apache.flink.configuration.GlobalConfiguration

ful so far. > > We recently changed a bit how we load the GlobalConfiguration in > combination with dynamic properties [1]. Maybe this has affected what > you've reported as well. > > [1] https://issues.apache.org/jira/browse/FLINK-7269 > > Cheers, > Till > > On Tue,

Re: Empty directories left over from checkpointing

>> delete. In the meantime, unfortunately the best way is to cleanup empty >> directories with some external tool. >> >> Best, >> Stefan >> >> Am 20.09.2017 um 01:23 schrieb Hao Sun <ha...@zendesk.com>: >> >> Thanks Elias! Seems like ther

Re: metrics for Flink sinks

Not an exact match, but I am guessing it is related to FLINK-7286 , which I reported. Feel free to modify that issue to cover the root cause. On Wed, Aug 30, 2017 at 8:32 AM, Martin Eden wrote: > Thanks

Re: can flink do streaming from data sources other than Kafka?

If you want to ensure you see all changes to a Cassandra table, you need to make use of the Change Data Capture feature. For that, you'll need code running on the Cassandra nodes to read the commit log segments from the Cassandra CDC

FLIP-17: Side Inputs

A bit late to this discussion, but I wanted to reiterate something that others also said. Side input readiness, and blocking until that is the case, is an important feature. This is specially true when the side input is used as a configuration stream. You don't want the main stream to be

FIP-6: Job specific Docker images status?

I was wondering what is the status of support for job specific Docker images, meaning images that combine the job jars with the job manager, do not require job submission, and automatically execute the job when there are enough task managers registered with the job manager to satisfy the job's

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

Why a range instead of just a single port in HA mode? On Mon, Sep 25, 2017 at 1:49 PM, Till Rohrmann wrote: > Yes, with Flip-6 it will most likely look like how Stephan described it. > We need the explicit port in standalone mode so that TMs can connect to the > JM. In the

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

sily lead to clashes if there is another > JobManager running on the same machine with the same port (e.g. due to > standby JobManagers). > > Cheers, > Till > > On Sep 26, 2017 03:20, "Elias Levy" <fearsome.lucid...@gmail.com> wrote: > >> Why a range inst

Global State and Scaling

I am implementing a control stream. The stream communicates a global configuration value for the whole job. It uses DataStream.broadcast() to communicate this to all parallel operator instances. I would like to save this value in state so that it can be recovered when the job restarts/recovers.

Re: Global State and Scaling

wse/FLINK-4940> for official support for broadcast state. Looks like the ugly hack is the way to go for now. On Mon, Aug 21, 2017 at 1:23 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I am implementing a control stream. The stream communicates a global > configuration

Prioritize DataStream

I believe the answer to this question is "no", but I figure I might as well ask. Is there a way to prioritize a stream? The use case is prioritization of a control stream. This is mostly needed on start-up, where a job might start consuming from the data stream before consuming from the control

Re: Prioritize DataStream

Flink folks, A response to the question below? On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I believe the answer to this question is "no", but I figure I might as > well ask. Is there a way to prioritize a stream? > > Th

Re: Kafka consumer are too fast for some partitions in "flatMap" like jobs

On Wed, Aug 30, 2017 at 11:50 AM, Oleksandr Baliev < aleksanderba...@gmail.com> wrote: > > So the main question is how to synchronize data reading between kafka > partitions when data is sequential per partitions, but late for some of > them and we care about that data is not thrown away and will

Re: Kafka consumer are too fast for some partitions in "flatMap" like jobs

How many partitions does the output topic have? If it has the same number of partitions as the input topic (30), have you considered simply using a custom partitioner for the Kafka sink that uses the input partition number as the output partition number? If the input messages are ordered per

FlinkKafkaProducer011 and Flink 1.4.0 Kafka docs

Looks like the Flink Kafka connector page, in the Producer section , is missing a section for the new FlinkKafkaProducer011 producer. Given that the new producer no longer has a static

Re: Flink flick cancel vs stop

by that by pushing data into a > (blocking) queue and having the "getElement()" method pull from that. > > On 15. Sep 2017, at 20:17, Elias Levy <fearsome.lucid...@gmail.com> wrote: > > On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright <eronwri...@gmail.com>

Incremental checkpointing documentation

There doesn't appear to be much in the way of documentation for incremental checkpointing other than how to turn it on. That leaves a lot of questions unanswered. What is the interaction of incremental checkpointing and external checkpoints? Any interaction with the

Odd job failure

We had a job on a Flink 1.4.2 cluster with three TMs experience an odd failure the other day. It seems that it started as some sort of network event. It began with the 3rd TM starting to warn every 30 seconds about socket timeouts while sending metrics to DataDog. This latest for the whole

Low Watermark: No Watermark

Any reason the web UI would show No Watermark for the Low Watermark of a job? The job is using a punctuated timestamp assigner with watermarks, and as far as I can tell it is generating watermarks properly.

Multiple stream operator watermark handling

Is there mechanism for a multiple stream operator to ignore watermarks from one of the streams? The use case is a multiple stream operator that consumes a primary stream and a secondary control stream. The control stream may only receive messages in rare occasion, and possibly never. The

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski wrote: > From top of my head I can imagine two solutions: > > 1. Override the default behaviour of the operator via for example > org.apache.flink.streaming.api.datastream.ConnectedStreams#transform > That seems the safer,

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 9:20 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski <pi...@data-artisans.com> > wrote: > >> From top of my head I can imagine two solutions: >> >> 1. Override the default be

Re: [DISCUSS] Flink 1.6 features

Since wishes are free: - Standalone cluster job isolation: https://issues.apache.org/jira/browse/FLINK-8886 - Proper sliding window joins (not overlapping hoping window joins): https://issues.apache.org/jira/browse/FLINK-6243 - Sharing state across operators:

Re: [DISCUSS] Flink 1.6 features

One more, since it we have to deal with it often: - Idling sources (Kafka in particular) and proper watermark propagation: FLINK-5018 / FLINK-5479 On Fri, Jun 8, 2018 at 2:58 PM, Elias Levy wrote: > Since wishes are free: > > - Standalone cluster job isolation: https://issues. >

Re: Odd job failure

On Mon, May 28, 2018 at 1:48 AM, Piotr Nowojski wrote: > Most likely suspect is the standard java problem of some dependency > convergence issue. Please check if you are not pulling in multiple Kafka > versions into your class path. Especially your job shouldn’t pull any Kafka > library except

Re: Odd job failure

case. >> >> 1. What Kafka version are you using? Have you looked for any known Kafka >> issues with those symptoms? >> 2. Maybe the easiest thing will be to reformat/reinstall/recreate TM3 AWS >> image? It might be some system issue. >> >> Piotrek >

high-availability.storageDir clean up?

I noticed in one of our cluster that they are relatively old submittedJobGraph* and completedCheckpoint* files. I was wondering at what point it is save to clean some of these up.

Cluster resurrects old job

We had an unusual situation last night. One of our Flink clusters experienced some connectivity issues, with lead to the the single job running on the cluster failing and then being restored. And then something odd happened. The cluster decided to also restore an old version of the job. One we

Re: Cluster resurrects old job

Alas, that error appears to be a red herring. Admin mistyped the cancel command leading to the error. But immediately corrected it, resulting in the job being canceled next. So seems unrelated to the job coming back to life later on. On Wed, Jun 20, 2018 at 10:04 AM Elias Levy wrote

Re: Cluster resurrects old job

) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) On Wed, Jun 20, 2018 at 9:31 AM Elias Levy wrote: > We had an unusual situation last night. One of our Flink clusters > experienced some connec

Description of Flink event time processing

The documentation of how Flink handles event time and watermarks is spread across several places. I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up. You can find it here:

Re: SQL materialized upsert tables

[ Adding the list back in, as this clarifies my question ] On Tue, Feb 20, 2018 at 3:42 PM, Darshan Singh wrote: > I am no expert in Flink but I will try my best. Issue you mentioned will > be with all streaming systems even with Kafka KTable I use them a lot for >

SQL materialized upsert tables

I noticed that has been significant work on the SQL / Table subsystem and decided to evaluate it for one of our use cases. The use case requires the joining of two streams, which can be considered a stream of table upserts. Critically, when joining the streams, we only want to join against the

Re: NoClassDefFoundError of a Avro class after cancel then resubmit the same job

I am currently suffering through similar issues. Had a job running happily, but when it the cluster tried to restarted it would not find the JSON serializer in it. The job kept trying to restart in a loop. Just today I was running a job I built locally. The job ran fine. I added two commits

Re: NoClassDefFoundError of a Avro class after cancel then resubmit the same job

Something seems to be off with the user code class loader. The only way I can get my job to start is if I drop the job into the lib folder in the JM and configure the JM's classloader.resolve-order to parent-first. Suggestions? On Thu, Feb 22, 2018 at 12:52 PM, Elias Levy <fearsome.lu

Re: SQL materialized upsert tables

On Wed, Feb 21, 2018 at 3:24 AM, Fabian Hueske wrote: > Hi Elias, > > Flink does not have built-in support for upsert stream -> table > conversions, yet. However, the community is working on that (see FLINK-8545 > [1]). > With a workaround, you can also solve the issue with

Flink network access control documentation

There is a need for better documentation on what connects to what over which ports in a Flink cluster to allow users to configure network access control rules. I was under the impression that in a ZK HA configuration the Job Managers were essentially independent and only coordinated via ZK. But

Old job resurrected during HA failover

For the second time in as many months we've had an old job resurrected during HA failover in a 1.4.2 standalone cluster. Failover was initiated when the leading JM lost its connection to ZK. I opened FLINK-10011 with the details. We are using

Re: Old job resurrected during HA failover

Vino, Thanks for the reply. Looking in ZK I see: [zk: localhost:2181(CONNECTED) 5] ls /flink/cluster_1/jobgraphs [d77948df92813a68ea6dfd6783f40e7e, 2a4eff355aef849c5ca37dbac04f2ff1] Again we see HA state for job 2a4eff355aef849c5ca37dbac04f2ff1, even though that job is no longer running (it

Re: Behavior of time based operators

See the section on Operators here https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing On Thu, Aug 2, 2018 at 3:42 PM Harshvardhan Agrawal < harshvardhan.ag...@gmail.com> wrote: > Hello, > > I have recently started reading Stream Processing with Apache

Re: Description of Flink event time processing

Fabian, https://github.com/apache/flink/pull/6481 I added a page, but did not remove or edit any existing page. Let me know what you'd like to see trimmed. On Thu, Aug 2, 2018 at 8:44 AM Fabian Hueske wrote: > Hi Elias, > > Thanks for the update! > I think the document can be added to the

Re: Old job resurrected during HA failover

Till, Thoughts? On Wed, Aug 1, 2018 at 7:34 PM vino yang wrote: > Your analysis is correct, yes, in theory the old jobgraph should be > deleted, but Flink currently uses the method of locking and asynchronously > deleting Path, so that it can not give you the acknowledgment of deleting, > so

Re: Description of Flink event time processing

Fabian, You have any time to review the changes? On Thu, Jul 19, 2018 at 2:19 AM Fabian Hueske wrote: > Hi Elias, > > Thanks for the update! > I'll try to have another look soon. > > Best, Fabian > > 2018-07-11 1:30 GMT+02:00 Elias Levy : > >> Thanks fo

Re: Counting elements that appear "behind" the watermark

You can create a ProcessFunction. That gives you access to getRuntimeContext to register metrics, to the element timestamp, and the current watermark. Keep in mind that operators first process a record and then process any watermark that was the result of that record, so that when you get the

Re: Counting elements that appear "behind" the watermark

Correct. Context gives you access to the element timestamp . But it also gives you access to the current watermark via timerService

Flink Forwards 2018 videos

It appears the Flink Forwards 2018 videos are FUBAR. The data entry form refuses to show them regardless of what you enter in it.

Re: Old job resurrected during HA failover

n't something call ZooKeeperStateHandleStore.releaseAll during HA failover to release the locks on the graphs? On Wed, Aug 1, 2018 at 9:49 AM Elias Levy wrote: > Thanks for the reply. Looking in ZK I see: > > [zk: localhost:2181(CONNECTED) 5] ls /flink/cluster

Re: Flink Rebalance

What do you consider a lot of latency? The rebalance will require serializing / deserializing the data as it gets distributed. Depending on the complexity of your records and the efficiency of your serializers, that could have a significant impact on your performance. On Thu, Aug 9, 2018 at

Why don't operations on KeyedStream return KeyedStream?

Operators on a KeyedStream don't return a new KeyedStream. Is there a reason for this? You need to perform `keyBy` again to get a KeyedStream. Presumably if you key by the same value there won't be any shuffled data, but the key may no longer be available within the stream record.

Queryable state and state TTL

Is there a reason queryable state can't work with state TTL? Trying to use both at the same time leads to a "IllegalArgumentException: Queryable state is currently not supported with TTL"

Re: StateMigrationException when switching from TypeInformation.of to createTypeInformation

Apologies for the delay. I've been traveling. On Mon, Jul 9, 2018 at 8:44 AM Till Rohrmann wrote: > could you check whether the `TypeInformation` returned by > `TypeInformation.of(new TypeHint[ConfigState]() {}))` and > `createTypeInformation[ConfigState]` return the same `TypeInformation` >

Re: event time and late events - documentation

Tovi, The document here should answer your question. If it doesn't, please let me know. On Mon, Jul 16, 2018 at 5:17 AM Sofer, Tovi wrote: > Hi group, > > Can someone please elaborate on the

Re: Implement Joins with Lookup Data

Alas, this suffer from the bootstrap problem. At the moment Flink does not allow you to pause a source (the positions), so you can't fully consume the and preload the accounts or products to perform the join before the positions start flowing. Additionally, Flink SQL does not support

Re: Description of Flink event time processing

Thanks for all the comments. I've updated the document to account for the feedback. Please take a look. On Fri, Jul 6, 2018 at 2:33 PM Elias Levy wrote: > Apologies. Comments are now enabled. > > On Thu, Jul 5, 2018 at 6:09 PM Rong Rong wrote: > >> Hi Elias, >&g

StateMigrationException when switching from TypeInformation.of to createTypeInformation

During some refactoring we changed a job using managed state from: ListStateDescriptor("config", TypeInformation.of(new TypeHint[ConfigState]() {})) to ListStateDescriptor("config", createTypeInformation[ConfigState]) After this change, Flink refused to start the new job from a savepoint or

Re: Description of Flink event time processing

for this write up! >>> I just skimmed your document and will provide more detailed feedback >>> later. >>> >>> It would be great to add such a page to the documentation. >>> >>> Best, Fabian >>> >>> 2018-07-03 3:07 GMT+02:00 El

Tracking deserialization errors

I was wondering how are folks tracking deserialization errors. The AbstractDeserializationSchema interface provides no mechanism for the deserializer to instantiate a metric counter, and "deserialize" must return a null instead of raising an exception in case of error if you want your job to

Re: Tracking deserialization errors

Either proposal would work. In the later case, at a minimum we'd need a way to identify the source within the metric. The basic error metric would then allow us to go into the logs to determine the cause of the error, as we already record the message causing trouble in the log. On Mon, Apr 16,

Re: String Interning

Am I the only one that feels the config should be renamed or the docs on it expanded? Turning on object reuse doesn't really reuse objects, not in the sense that an object can be reused for different values / messages / records. Instead, it stops Flink from making copies of of a record, by

Re: RocksDB checkpointing dir per TM

There is also state.backend.rocksdb.localdir. Oddly, I can find the documentation for it in the 1.5 docs , but not in the 1.6 docs

Re: Live configuration change

Also note that there is a pending PR to allow the Cassandra sink to back pressure, so that the cluster does not get overwhelmed. On Tue, Nov 6, 2018 at 12:46 PM Ning Shi wrote: > > for rate limiting, would quota at Kafka brokers help? > > Thanks, Steven. This looks very promising. I'll try it

Re: Kafka Per-Partition Watermarks

Does your job perform a keyBy or broadcast that would result in data from different partitions being distributed among tasks? If so, then that would be the cause. On Thu, Oct 4, 2018 at 12:58 PM Andrew Kowpak wrote: > Hi all, > > I apologize if this has been discussed to death in the past,

Scala case class state evolution

Currently it is impossible to evolve a Scala case class by adding a new field to it that is stored as managed state using the default Flink serializer and restore a the job from a savepoint created using the previous version of the class, correct?

Re: Deserialization of serializer errored

stable class name if the code is refactored, leading to the class no longer being found by that name in a new version of the job. On Tue, Oct 2, 2018 at 4:55 PM Elias Levy wrote: > To add to the mystery, I extracted the class file mentioned in the > exceptions (TestJob$$anon$13$$anon$3) from

Re: Deserialization of serializer errored

That would explain why you see an exception when the case class is > modified or completely removed. > > Maybe Stefan or Gordon can help here. > > Best, Fabian > > Am Di., 2. Okt. 2018 um 01:10 Uhr schrieb Elias Levy < > fearsome.lucid...@gmail.com>: > >> Any

Deserialization of serializer errored

I am experiencing a rather odd error. We have a job running on a Flink 1.4.2 cluster with two Kafka input streams, one of the streams is processed by an async function, and the output of the async function and the other original stream are consumed by a CoProcessOperator, that intern emits Scala

  1   2   >