Remote Stateful Function Scalability

After reading the Stateful Functions documentation, I am left wondering how remote stateful functions scale. The documentation mentions that the use of remote functions allows the state and compute tiers to scale independently. But the documentation seems to imply that only a single instance of a

Re: Kafka Schema registry

Just for a Kafka source: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#the-deserializationschema - There is also a version of this schema available that can lookup the writer’s schema (schema which was used to write the record) in Confluent Schema

Re: Scylla connector

Scylla is protocol compatible with Cassandra, so you can just use the Cassandra connector. Scylla has extended the Go gocql package to make it shard aware, but such an extension does not exist for the Cassandra Java driver. That just means that the driver will sent requests to any shard on a

Re: Does Flink Kafka connector has max_pending_offsets concept?

There is no such concept in Flink. Flink tracks offsets in its checkpoints. It can optionally commit offsets to Kafka, but that is only for reporting purposes. If you wish to lower the number of records that get reprocessed in the case of a restart, then you must lower the checkpoint interval.

Re: Flink vs KStreams

My 2c: KStreams: Pros: * Streaming as a library: No need to submit your job to a cluster. Easy to scale up/down the job by adding or removing workers. * Streaming durability: State is durably stored in Kafka topics in a streaming fashion. Durability is amortized across the job's lifetime. * No

Re: [Discuss] Semantics of event time for state TTL

e: processing-time >> >> What do you think? >> >> Aljoscha >> >> > On 6. Apr 2019, at 01:30, Konstantin Knauf >> wrote: >> > >> > Hi Andrey, >> > >> > I agree with Elias. This would be the most natural behavior. I wouldn

Re: [Discuss] Semantics of event time for state TTL

My 2c: Timestamp stored with the state value: Event timestamp Timestamp used to check expiration: Last emitted watermark That follows the event time processing model used elsewhere is Flink. E.g. events are segregated into windows based on their event time, but the windows do not fire until the

Re: What should I take care if I enable object reuse

That's certainly the safe thing to do, but if you do not mutate the object, a copy is not strictly necessary. On Thu, Mar 14, 2019 at 9:19 PM Kurt Young wrote: > Keep one thing in mind: if you want the element remains legal after the > function call ends (maybe map(), flatmap(), depends on

Re: using updating shared data

ffectively be ignored. >> >> On Thu, Jan 3, 2019 at 9:18 PM Avi Levi wrote: >> >>> Thanks for the tip Elias! >>> >>> On Wed, Jan 2, 2019 at 9:44 PM Elias Levy >>> wrote: >>> >>>> One thing you must be careful of, is that if yo

Re: using updating shared data

One thing you must be careful of, is that if you are using event time processing, assuming that the control stream will only receive messages sporadically, is that event time will stop moving forward in the operator joining the streams while the control stream is idle. You can get around this by

Re: Live configuration change

Also note that there is a pending PR to allow the Cassandra sink to back pressure, so that the cluster does not get overwhelmed. On Tue, Nov 6, 2018 at 12:46 PM Ning Shi wrote: > > for rate limiting, would quota at Kafka brokers help? > > Thanks, Steven. This looks very promising. I'll try it

Re: RocksDB checkpointing dir per TM

There is also state.backend.rocksdb.localdir. Oddly, I can find the documentation for it in the 1.5 docs , but not in the 1.6 docs

Re: Watermark on keyed stream

You are correct that watermarks are not tracked per key. You are dealing with events with a high degree of delay variability. That is usually not a good match for event time processing as implemented in Flink. You could use event time processing and configure a very large window allowed

Re: Kafka Per-Partition Watermarks

Does your job perform a keyBy or broadcast that would result in data from different partitions being distributed among tasks? If so, then that would be the cause. On Thu, Oct 4, 2018 at 12:58 PM Andrew Kowpak wrote: > Hi all, > > I apologize if this has been discussed to death in the past,

Re: Deserialization of serializer errored

stable class name if the code is refactored, leading to the class no longer being found by that name in a new version of the job. On Tue, Oct 2, 2018 at 4:55 PM Elias Levy wrote: > To add to the mystery, I extracted the class file mentioned in the > exceptions (TestJob$$anon$13$$anon$3) from

Re: Deserialization of serializer errored

That would explain why you see an exception when the case class is > modified or completely removed. > > Maybe Stefan or Gordon can help here. > > Best, Fabian > > Am Di., 2. Okt. 2018 um 01:10 Uhr schrieb Elias Levy < > fearsome.lucid...@gmail.com>: > >> Any

Scala case class state evolution

Currently it is impossible to evolve a Scala case class by adding a new field to it that is stored as managed state using the default Flink serializer and restore a the job from a savepoint created using the previous version of the class, correct?

Re: Deserialization of serializer errored

Any of the Flink folks seen this before? On Fri, Sep 28, 2018 at 5:23 PM Elias Levy wrote: > I am experiencing a rather odd error. We have a job running on a Flink > 1.4.2 cluster with two Kafka input streams, one of the streams is processed > by an async function, and the output of

Deserialization of serializer errored

I am experiencing a rather odd error. We have a job running on a Flink 1.4.2 cluster with two Kafka input streams, one of the streams is processed by an async function, and the output of the async function and the other original stream are consumed by a CoProcessOperator, that intern emits Scala

Queryable state and state TTL

Is there a reason queryable state can't work with state TTL? Trying to use both at the same time leads to a "IllegalArgumentException: Queryable state is currently not supported with TTL"

Why don't operations on KeyedStream return KeyedStream?

Operators on a KeyedStream don't return a new KeyedStream. Is there a reason for this? You need to perform `keyBy` again to get a KeyedStream. Presumably if you key by the same value there won't be any shuffled data, but the key may no longer be available within the stream record.

Re: Flink Rebalance

What do you consider a lot of latency? The rebalance will require serializing / deserializing the data as it gets distributed. Depending on the complexity of your records and the efficiency of your serializers, that could have a significant impact on your performance. On Thu, Aug 9, 2018 at

Flink Forwards 2018 videos

It appears the Flink Forwards 2018 videos are FUBAR. The data entry form refuses to show them regardless of what you enter in it.

Re: Old job resurrected during HA failover

Till, Thoughts? On Wed, Aug 1, 2018 at 7:34 PM vino yang wrote: > Your analysis is correct, yes, in theory the old jobgraph should be > deleted, but Flink currently uses the method of locking and asynchronously > deleting Path, so that it can not give you the acknowledgment of deleting, > so

Re: Behavior of time based operators

See the section on Operators here https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing On Thu, Aug 2, 2018 at 3:42 PM Harshvardhan Agrawal < harshvardhan.ag...@gmail.com> wrote: > Hello, > > I have recently started reading Stream Processing with Apache

Re: Description of Flink event time processing

Fabian, https://github.com/apache/flink/pull/6481 I added a page, but did not remove or edit any existing page. Let me know what you'd like to see trimmed. On Thu, Aug 2, 2018 at 8:44 AM Fabian Hueske wrote: > Hi Elias, > > Thanks for the update! > I think the document can be added to the

Re: Old job resurrected during HA failover

n't something call ZooKeeperStateHandleStore.releaseAll during HA failover to release the locks on the graphs? On Wed, Aug 1, 2018 at 9:49 AM Elias Levy wrote: > Thanks for the reply. Looking in ZK I see: > > [zk: localhost:2181(CONNECTED) 5] ls /flink/cluster

Re: Old job resurrected during HA failover

Vino, Thanks for the reply. Looking in ZK I see: [zk: localhost:2181(CONNECTED) 5] ls /flink/cluster_1/jobgraphs [d77948df92813a68ea6dfd6783f40e7e, 2a4eff355aef849c5ca37dbac04f2ff1] Again we see HA state for job 2a4eff355aef849c5ca37dbac04f2ff1, even though that job is no longer running (it

Old job resurrected during HA failover

For the second time in as many months we've had an old job resurrected during HA failover in a 1.4.2 standalone cluster. Failover was initiated when the leading JM lost its connection to ZK. I opened FLINK-10011 with the details. We are using

Re: Counting elements that appear "behind" the watermark

Correct. Context gives you access to the element timestamp . But it also gives you access to the current watermark via timerService

Re: Counting elements that appear "behind" the watermark

You can create a ProcessFunction. That gives you access to getRuntimeContext to register metrics, to the element timestamp, and the current watermark. Keep in mind that operators first process a record and then process any watermark that was the result of that record, so that when you get the

Re: Description of Flink event time processing

Fabian, You have any time to review the changes? On Thu, Jul 19, 2018 at 2:19 AM Fabian Hueske wrote: > Hi Elias, > > Thanks for the update! > I'll try to have another look soon. > > Best, Fabian > > 2018-07-11 1:30 GMT+02:00 Elias Levy : > >> Thanks fo

Re: Implement Joins with Lookup Data

Alas, this suffer from the bootstrap problem. At the moment Flink does not allow you to pause a source (the positions), so you can't fully consume the and preload the accounts or products to perform the join before the positions start flowing. Additionally, Flink SQL does not support

Re: event time and late events - documentation

Tovi, The document here should answer your question. If it doesn't, please let me know. On Mon, Jul 16, 2018 at 5:17 AM Sofer, Tovi wrote: > Hi group, > > Can someone please elaborate on the

Re: StateMigrationException when switching from TypeInformation.of to createTypeInformation

Apologies for the delay. I've been traveling. On Mon, Jul 9, 2018 at 8:44 AM Till Rohrmann wrote: > could you check whether the `TypeInformation` returned by > `TypeInformation.of(new TypeHint[ConfigState]() {}))` and > `createTypeInformation[ConfigState]` return the same `TypeInformation` >

Re: Description of Flink event time processing

Thanks for all the comments. I've updated the document to account for the feedback. Please take a look. On Fri, Jul 6, 2018 at 2:33 PM Elias Levy wrote: > Apologies. Comments are now enabled. > > On Thu, Jul 5, 2018 at 6:09 PM Rong Rong wrote: > >> Hi Elias, >&g

Re: Description of Flink event time processing

for this write up! >>> I just skimmed your document and will provide more detailed feedback >>> later. >>> >>> It would be great to add such a page to the documentation. >>> >>> Best, Fabian >>> >>> 2018-07-03 3:07 GMT+02:00 El

StateMigrationException when switching from TypeInformation.of to createTypeInformation

During some refactoring we changed a job using managed state from: ListStateDescriptor("config", TypeInformation.of(new TypeHint[ConfigState]() {})) to ListStateDescriptor("config", createTypeInformation[ConfigState]) After this change, Flink refused to start the new job from a savepoint or

Description of Flink event time processing

The documentation of how Flink handles event time and watermarks is spread across several places. I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up. You can find it here:

Re: String Interning

Am I the only one that feels the config should be renamed or the docs on it expanded? Turning on object reuse doesn't really reuse objects, not in the sense that an object can be reused for different values / messages / records. Instead, it stops Flink from making copies of of a record, by

high-availability.storageDir clean up?

I noticed in one of our cluster that they are relatively old submittedJobGraph* and completedCheckpoint* files. I was wondering at what point it is save to clean some of these up.

Re: Cluster resurrects old job

Alas, that error appears to be a red herring. Admin mistyped the cancel command leading to the error. But immediately corrected it, resulting in the job being canceled next. So seems unrelated to the job coming back to life later on. On Wed, Jun 20, 2018 at 10:04 AM Elias Levy wrote

Re: Cluster resurrects old job

) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) On Wed, Jun 20, 2018 at 9:31 AM Elias Levy wrote: > We had an unusual situation last night. One of our Flink clusters > experienced some connec

Cluster resurrects old job

We had an unusual situation last night. One of our Flink clusters experienced some connectivity issues, with lead to the the single job running on the cluster failing and then being restored. And then something odd happened. The cluster decided to also restore an old version of the job. One we

Re: [DISCUSS] Flink 1.6 features

One more, since it we have to deal with it often: - Idling sources (Kafka in particular) and proper watermark propagation: FLINK-5018 / FLINK-5479 On Fri, Jun 8, 2018 at 2:58 PM, Elias Levy wrote: > Since wishes are free: > > - Standalone cluster job isolation: https://issues. >

Re: [DISCUSS] Flink 1.6 features

Since wishes are free: - Standalone cluster job isolation: https://issues.apache.org/jira/browse/FLINK-8886 - Proper sliding window joins (not overlapping hoping window joins): https://issues.apache.org/jira/browse/FLINK-6243 - Sharing state across operators:

Re: Odd job failure

On Mon, May 28, 2018 at 1:48 AM, Piotr Nowojski wrote: > Most likely suspect is the standard java problem of some dependency > convergence issue. Please check if you are not pulling in multiple Kafka > versions into your class path. Especially your job shouldn’t pull any Kafka > library except

Re: Odd job failure

case. >> >> 1. What Kafka version are you using? Have you looked for any known Kafka >> issues with those symptoms? >> 2. Maybe the easiest thing will be to reformat/reinstall/recreate TM3 AWS >> image? It might be some system issue. >> >> Piotrek >

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 9:20 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski <pi...@data-artisans.com> > wrote: > >> From top of my head I can imagine two solutions: >> >> 1. Override the default be

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski wrote: > From top of my head I can imagine two solutions: > > 1. Override the default behaviour of the operator via for example > org.apache.flink.streaming.api.datastream.ConnectedStreams#transform > That seems the safer,

Multiple stream operator watermark handling

Is there mechanism for a multiple stream operator to ignore watermarks from one of the streams? The use case is a multiple stream operator that consumes a primary stream and a secondary control stream. The control stream may only receive messages in rare occasion, and possibly never. The

Low Watermark: No Watermark

Any reason the web UI would show No Watermark for the Low Watermark of a job? The job is using a punctuated timestamp assigner with watermarks, and as far as I can tell it is generating watermarks properly.

Odd job failure

We had a job on a Flink 1.4.2 cluster with three TMs experience an odd failure the other day. It seems that it started as some sort of network event. It began with the 3rd TM starting to warn every 30 seconds about socket timeouts while sending metrics to DataDog. This latest for the whole

Re: Tracking deserialization errors

Either proposal would work. In the later case, at a minimum we'd need a way to identify the source within the metric. The basic error metric would then allow us to go into the logs to determine the cause of the error, as we already record the message causing trouble in the log. On Mon, Apr 16,

Tracking deserialization errors

I was wondering how are folks tracking deserialization errors. The AbstractDeserializationSchema interface provides no mechanism for the deserializer to instantiate a metric counter, and "deserialize" must return a null instead of raising an exception in case of error if you want your job to

Re: NoClassDefFoundError of a Avro class after cancel then resubmit the same job

Something seems to be off with the user code class loader. The only way I can get my job to start is if I drop the job into the lib folder in the JM and configure the JM's classloader.resolve-order to parent-first. Suggestions? On Thu, Feb 22, 2018 at 12:52 PM, Elias Levy <fearsome.lu

Re: NoClassDefFoundError of a Avro class after cancel then resubmit the same job

I am currently suffering through similar issues. Had a job running happily, but when it the cluster tried to restarted it would not find the JSON serializer in it. The job kept trying to restart in a loop. Just today I was running a job I built locally. The job ran fine. I added two commits

Re: SQL materialized upsert tables

On Wed, Feb 21, 2018 at 3:24 AM, Fabian Hueske wrote: > Hi Elias, > > Flink does not have built-in support for upsert stream -> table > conversions, yet. However, the community is working on that (see FLINK-8545 > [1]). > With a workaround, you can also solve the issue with

Re: SQL materialized upsert tables

[ Adding the list back in, as this clarifies my question ] On Tue, Feb 20, 2018 at 3:42 PM, Darshan Singh wrote: > I am no expert in Flink but I will try my best. Issue you mentioned will > be with all streaming systems even with Kafka KTable I use them a lot for >

SQL materialized upsert tables

I noticed that has been significant work on the SQL / Table subsystem and decided to evaluate it for one of our use cases. The use case requires the joining of two streams, which can be considered a stream of table upserts. Critically, when joining the streams, we only want to join against the

Flink network access control documentation

There is a need for better documentation on what connects to what over which ports in a Flink cluster to allow users to configure network access control rules. I was under the impression that in a ZK HA configuration the Job Managers were essentially independent and only coordinated via ZK. But

Re: Flink flick cancel vs stop

by that by pushing data into a > (blocking) queue and having the "getElement()" method pull from that. > > On 15. Sep 2017, at 20:17, Elias Levy <fearsome.lucid...@gmail.com> wrote: > > On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright <eronwri...@gmail.com>

FlinkKafkaProducer011 and Flink 1.4.0 Kafka docs

Looks like the Flink Kafka connector page, in the Producer section , is missing a section for the new FlinkKafkaProducer011 producer. Given that the new producer no longer has a static

Incremental checkpointing documentation

There doesn't appear to be much in the way of documentation for incremental checkpointing other than how to turn it on. That leaves a lot of questions unanswered. What is the interaction of incremental checkpointing and external checkpoints? Any interaction with the

Re: Empty directories left over from checkpointing

>> delete. In the meantime, unfortunately the best way is to cleanup empty >> directories with some external tool. >> >> Best, >> Stefan >> >> Am 20.09.2017 um 01:23 schrieb Hao Sun <ha...@zendesk.com>: >> >> Thanks Elias! Seems like ther

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

sily lead to clashes if there is another > JobManager running on the same machine with the same port (e.g. due to > standby JobManagers). > > Cheers, > Till > > On Sep 26, 2017 03:20, "Elias Levy" <fearsome.lucid...@gmail.com> wrote: > >> Why a range inst

FIP-6: Job specific Docker images status?

I was wondering what is the status of support for job specific Docker images, meaning images that combine the job jars with the job manager, do not require job submission, and automatically execute the job when there are enough task managers registered with the job manager to satisfy the job's

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

Why a range instead of just a single port in HA mode? On Mon, Sep 25, 2017 at 1:49 PM, Till Rohrmann wrote: > Yes, with Flip-6 it will most likely look like how Stephan described it. > We need the explicit port in standalone mode so that TMs can connect to the > JM. In the

high-availability.jobmanager.port vs jobmanager.rpc.port

I am wondering why HA mode there is a need for a separate config parameter to set the JM RPC port (high-availability.jobmanager.port) and why this parameter accepts a range, unlike jobmanager.rpc.port.

History Server

I am curious, why is the History Server a separate process and Web UI instead of being part of the Web Dashboard within the Job Manager?

Re: Noisy org.apache.flink.configuration.GlobalConfiguration

ful so far. > > We recently changed a bit how we load the GlobalConfiguration in > combination with dynamic properties [1]. Maybe this has affected what > you've reported as well. > > [1] https://issues.apache.org/jira/browse/FLINK-7269 > > Cheers, > Till > > On Tue,

Re: Flink flick cancel vs stop

On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright wrote: > Aljoscha, would it be correct to characterize your idea as a 'pull' source > rather than the current 'push'? It would be interesting to look at the > existing connectors to see how hard it would be to reverse their >

Re: Assigning operators to slots

The execution within the IDE is most likely not loading the flink-conf.yaml file to read the configuration. When run from the IDE you get a LocalStreamEnvironment, which starts a LocalFlinkMiniCluster. LocalStreamEnvironment is created by StreamExecutionEnvironment.createLocalEnvironment without

Re: Flink flick cancel vs stop

Anyone? On Mon, Sep 11, 2017 at 6:17 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I was wondering about the status of the flink stop command. At first > blush it would seem as the preferable way to shutdown a Flink job, but it > depends on StoppableFunction being impleme

Flink flick cancel vs stop

I was wondering about the status of the flink stop command. At first blush it would seem as the preferable way to shutdown a Flink job, but it depends on StoppableFunction being implemented by sources and I notice that the Kafka source does not seem to implement it. In addition, the command does

FLIP-17: Side Inputs

A bit late to this discussion, but I wanted to reiterate something that others also said. Side input readiness, and blocking until that is the case, is an important feature. This is specially true when the side input is used as a configuration stream. You don't want the main stream to be

Re: can flink do streaming from data sources other than Kafka?

If you want to ensure you see all changes to a Cassandra table, you need to make use of the Change Data Capture feature. For that, you'll need code running on the Cassandra nodes to read the commit log segments from the Cassandra CDC

Re: Kafka consumer are too fast for some partitions in "flatMap" like jobs

On Wed, Aug 30, 2017 at 11:50 AM, Oleksandr Baliev < aleksanderba...@gmail.com> wrote: > > So the main question is how to synchronize data reading between kafka > partitions when data is sequential per partitions, but late for some of > them and we care about that data is not thrown away and will

Re: metrics for Flink sinks

Not an exact match, but I am guessing it is related to FLINK-7286 , which I reported. Feel free to modify that issue to cover the root cause. On Wed, Aug 30, 2017 at 8:32 AM, Martin Eden wrote: > Thanks

Re: Kafka consumer are too fast for some partitions in "flatMap" like jobs

How many partitions does the output topic have? If it has the same number of partitions as the input topic (30), have you considered simply using a custom partitioner for the Kafka sink that uses the input partition number as the output partition number? If the input messages are ordered per

Re: Global State and Scaling

wse/FLINK-4940> for official support for broadcast state. Looks like the ugly hack is the way to go for now. On Mon, Aug 21, 2017 at 1:23 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I am implementing a control stream. The stream communicates a global > configuration

Global State and Scaling

I am implementing a control stream. The stream communicates a global configuration value for the whole job. It uses DataStream.broadcast() to communicate this to all parallel operator instances. I would like to save this value in state so that it can be recovered when the job restarts/recovers.

Re: Prioritize DataStream

Flink folks, A response to the question below? On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I believe the answer to this question is "no", but I figure I might as > well ask. Is there a way to prioritize a stream? > > Th

Prioritize DataStream

I believe the answer to this question is "no", but I figure I might as well ask. Is there a way to prioritize a stream? The use case is prioritization of a control stream. This is mostly needed on start-up, where a job might start consuming from the data stream before consuming from the control

Re: CEP memory requirements

Looking at the code I gather that 1.2 does not clear the per key NFA state even if there is no state to keep, whereas this appears fixed in the master branch. Yes? On Thu, May 4, 2017 at 11:25 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > I am observing odd memory behavior wit

CEP memory requirements

I am observing odd memory behavior with the CEP library and I am wondering if it is expected. If I write a simple local streaming Flink job that reads from a 65MB compressed file of JSON objects, one per line, parses the JSON, performs a filter operation, and then a keyBy, heap usage is stable,

Re: RocksDB error with flink 1.2.0

Any reason they can't share a single RocksDB state backend instance? On Fri, Apr 28, 2017 at 8:44 AM, Aljoscha Krettek wrote: > The problem here is that this will try to open 300 RocksDB instances on > each of the TMs (depending on how the parallelism is spread between the

Re: CEP join across events

It would be useful if there were a cleaner syntax for specifying relationships between matched events, as in an SQL join, particularly for conditions with a quantifier of one. At the moment you have to do something like Pattern. .begin[Foo]("first") .where( first => first.baz

Re: CEP join across events

ched > in "foo" pattern in that particular branch. > I mean that for a sequence like (type =1, value_b = X); (type=1, > value_b=Y); (type=2, value_b=X) both events of type = 1 create a seperate > pattern branch and the event with type = 2 will be checked for a matc

Re: Re-keying / sub-keying a stream without repartitioning

On Wed, Apr 26, 2017 at 5:11 AM, Aljoscha Krettek wrote: > I did spend some time thinking about this and we had the idea for a while > now to add an operation like “keyByWithoutPartitioning()” (name not final > ;-) that would allow the user to tell the system that we don’t

CEP join across events

There doesn't appear to be a way to join events across conditions using the CEP library. Consider events of the form (type, value_a, value_b) on a stream keyed by the value_a field. Under 1.2 you can create a pattern that for a given value_a, as specified by the stream key, there is a match if

Re: Re-keying / sub-keying a stream without repartitioning

Anyone? On Fri, Apr 21, 2017 at 10:15 PM, Elias Levy <fearsome.lucid...@gmail.com> wrote: > This is something that has come up before on the list, but in a different > context. I have a need to rekey a stream but would prefer the stream to > not be repartitioned. T

Re-keying / sub-keying a stream without repartitioning

This is something that has come up before on the list, but in a different context. I have a need to rekey a stream but would prefer the stream to not be repartitioned. There is no gain to repartitioning, as the new partition key is a composite of the stream key, going from a key of A to a key of

Does Flink DataStreams using combiners?

I am wondering if Flink makes use of combiners to pre-reduce a keyed and windowed stream before shuffling the data among workers. I.e. will it use a combiner in something like: stream.flatMap {...} .assignTimestampsAndWatermarks(...) .keyBy(...) .timeWindow(...)

Arrays values in keyBy

I would be useful if the documentation warned what type of equality it expected of values used as keys in keyBy. I just got bit in the ass by converting a field from a string to a byte array. All of the sudden the windows were no longer aggregating. So it seems Flink is not doing a deep compare

Re: FlinkKafkaProducer API

On Thu, Jun 9, 2016 at 5:16 AM, Fabian Hueske wrote: > thanks for your feedback. I think those are good observations and > suggestions to improve the Kafka producers. > The best place to discuss such improvements is the dev mailing list. > > Would like to repost your mail

FlinkKafkaProducer API

The FlinkKafkaProducer API seems more difficult to use than it should be. The API requires you pass it a SerializationSchema or a KeyedSerializationSchema, but the Kafka producer already has a serialization API. Requiring a serializer in the Flink API precludes the use of the Kafka serializers.

Re: Kafka producer sink message loss?

On Tue, Jun 7, 2016 at 4:52 AM, Stephan Ewen wrote: > The concern you raised about the sink being synchronous is exactly what my > last suggestion should address: > > The internal state backends can return a handle that can do the sync in a > background thread. The sink would

Re: Event processing time with lateness

On Fri, Jun 3, 2016 at 6:47 AM, Kostas Kloudas wrote: > To see a relatively more complex example of such a trigger and how to > implement it, > you can have a look at this implementation: >

Re: Kafka producer sink message loss?

On Sun, Jun 5, 2016 at 3:16 PM, Stephan Ewen wrote: > You raised a good point. Fortunately, there should be a simply way to fix > this. > > The Kafka Sunk Function should implement the "Checkpointed" interface. It > will get a call to the "snapshotState()" method whenever a

  1   2   >