Re: How to clear keyed states periodically?

Hi vino, Thanks for the advice, but I think state TTL does not completely fit in my case. AFAIK, State TTL is per entry level and uses an inactive time threshold to expire entries, but I need a TTL for the whole MapState, which does not depend on when the entries are created or updated. Suppose

Re: ListState - elements order

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge. I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default. -- Rong On Thu, Sep 13, 2018 at 7:30 PM vino yang wrote: > Hi Aljoscha, > > Regard

Re: Client failed to get cancel with savepoint response(Internet mail)

Hi vino, Thank you for the helpful information! One more question, are these operations supposed to run concurrently to ensure JobManager receives the cancel request before the savepoint is completed? Best, Paul lam > 在 2018年9月14日，11:48，vino yang 写道： > > Hi Paul, > > It does not affect any

Re: Watermarks in Event Time windowing

Yes in many cases what we have faced that let's say in a keyed stream an element of a specific key comes in which triggers a new window. If a corresponding elements of the same key does not arrive a new watermark is not generated for the window to purge. Then we faced issues with flink keeping reco

Re: Client failed to get cancel with savepoint response(Internet mail)

Hi Paul, It does not affect anything. It simply splits the two operations that can be connected together into two separate operations. The cancel operation will not be triggered until the savepoint operation is completed. Thanks, vino. Paul Lam 于2018年9月14日周五 上午11:12写道： > Hi Devin, > > Thanks f

Re: Client failed to get cancel with savepoint response(Internet mail)

Hi Devin, Thanks for the reply! It seems like I missed an important thread. @vino mentioned a solution that is splitting the cancel-with-savepoint operation into two separated operations, and I wonder if it breaks the end to end exactly-once semantics in case of a at-least-once sink? Thanks a

回复: Client failed to get cancel with savepoint response(Internet mail)

Hi, Paul https://issues.apache.org/jira/browse/FLINK-10309 发件人： Paul Lam 发送时间： 2018-09-14 10:41 收件人： user 主题： Client failed to get cancel with savepoint response(Internet mail) Hi, I've been experiencing an issue that the client fro

Client failed to get cancel with savepoint response

Hi, I've been experiencing an issue that the client frontend is likely unable to get cancel-with-savepoint responses from JobManager sometimes. I’m using Flink 1.5.3 on YARN cluster, and when I execute cancel with savepoint command (FsStataBackend backed by HDFS), normally it finishes in seco

Re: How to clear keyed states periodically?

Hi Paul, Maybe you can try to understand the State TTL?[1] Thanks, vino. [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#state-time-to-live-ttl Paul Lam 于2018年9月12日周三 下午6:06写道： > Hi, > > I’m using MapState to deduplicate some ids and the MapState n

Re: Task managers run on separate nodes in a cluster

Hi Martin, Till has done most of the work of Flink on Mesos. Ping Till for you. Thanks, vino. Martin Eden 于2018年9月12日周三 下午11:21写道： > Hi all, > > We're using Flink 1.3.2 with DCOS / Mesos. > > We have a 3 node cluster and are running the Flink DCOS package (Flink > Mesos framework) configured w

Re: Custom metrics in Watermark Assigner

Hi Oleksandr, It sounds like a good idea to make a lot of internal information more transparent. But it would be better if you could be more specific. Thanks, vino. Oleksandr Nitavskyi 于2018年9月12日周三 下午9:48写道： > Hello guys, > > > > In our custom AssignerWithPunctuatedWatermarks we want to have

Re: ListState - elements order

Hi Aljoscha, Regarding window merging, as you said, it's not clear, because Flink does some internal work. But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it i

Re: Question Regarding Streaming pipeline

Hi bhasker, Currently, using the Jython?Flink Python API in the JVM ecosystem is also based on Jython. Maybe you use Apache Beam's Python API to combine with Flink runner is a better choice. But maybe Chesnay, can give you better advice. Ping Chesnay for you. Thanks, vino. bhaskar.eba...@gmail.c

Re: Watermarks in Event Time windowing

Hi Taher, For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1] About this question: What happens if the watermark is same as the timestamp? Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an id

Re: Flink Checkpointing in production

Hi Ahmad, Answer your question： Ahmad Hassan 于2018年9月12日周三 下午4:39写道： > Hi All, > > We need two clarifications for using Flink 1.6.0. We have flink jobs > running to handle 100's of tenants with sliding window of 24hrs and slide > by 5 minutes. > > 1) If checkpointing is enabled and flink job cr

Re: Can rocksDBBackend handle rescaling?

Thanks for your answer, but I still have some questions. Could you tell me why the checkpoint cannot be used for rescaling? From my perspective, the difference between checkpoint and savepoint is nullable externalSavepointLocation. And from this doc:https://flink.apache.org/features/2017/07/04/f

Watermarks in Event Time windowing

Hi All, Can someone show a good example of how watermarks need to be generated when using EventTime windows? What happens if the watermark is same as the timestamp? How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in

Re: Can rocksDBBackend handle rescaling?

A checkpoint cannot be used for rescaling. You should take a savepoint, stop the application, and resume from the savepoint with a new parallelism. https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html On Fri, Sep 14, 2018 at 1:50 AM 廖嘉逸 wrote: > Hi, all > >

Can rocksDBBackend handle rescaling?

Hi, all I’m using Flink 1.6 and I’m goint to use RocksDBBackend as the checkpoint state backend. Before that, I need to make sure that the checkpoint can handle rescale. From the docs on Apache Flink https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html, it

Question Regarding Streaming pipeline

Hi In the streaming pipe line of Scala we want to use intermediary Machine Learning module written in python, Is it possible to use it ? Or entire pipe line should be either Python or Scala, we can't mix? Regards Bhaskar

Re: Weird behaviour after change sources in a job.

Great ! So I have created a ticket: https://issues.apache.org/jira/browse/FLINK-10342 with a test which reproduces the issue: https://github.com/apache/flink/pull/6691/files If it seems reasonable I can create a fix for this. Regards Oleksandr From: Dawid Wysakowicz Date: Thursday, 13 Septemb

Re: ListState - elements order

Hi, this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge. Best,

Re: [Kerberos] JAAS module content not generated? javax.security.auth.callback.UnsupportedCallbackException: Could not login: the client is being asked for a password, but the Kafka client code does n

Hi Sebastien, I don't think empty jaas.conf file is a problem here. This file is created just to satisfy some preconditions which check for existence of that file. The actual values are passed in memory. Usually when I had problems with kerberos it was due to hostname checks. Make sure you use FQ

Re: [Kerberos] JAAS module content not generated? javax.security.auth.callback.UnsupportedCallbackException: Could not login: the client is being asked for a password, but the Kafka client code does n

Hi, this definitely needs some investigation! Did you try setting the JAAS settings in the sasl.jaas.config property as described in https://docs.confluent.io/current/kafka/authentication_sasl/index.html#jaas-configurations?

Re: Filescheme GS not found sometimes - inconsistent exceptions for reading from GCS

Sorry, I had a second look and your stacktrace does not even point to the spilling channel - it reads from the memory segment directly. -> setting the temp dirs will thus not make a difference I'm wondering why your deserializer eventually reads from a file on gs:// directly, instead of, for examp

Re: StreamingFileSink with PrestoFS?

Aha, thanks! On Thu, Sep 13, 2018 at 2:53 PM, Aljoscha Krettek wrote: > Hi Jared, > > using the new StreamingFileSink is indeed not supported for S3 (PrestoFS > or not) right now. Work on this is tracked under https://issues.apache. > org/jira/browse/FLINK-9752 and should hopefully make it into

Re: Test harness for validating proper checkpointing of custom SourceFunction

Hi Ken, you can use the (slightly misnamed for this purpose) AbstractStreamOperatorTestHarness. It's used in the Flink codebase for unit testing sources. Best, Aljoscha > On 12. Sep 2018, at 23:37, Ken Krugler wrote: > > Hi all, > > We’re using the (Keyed)(One|Two)InputStreamOperatorTestHar

Re: StreamingFileSink with PrestoFS?

Hi Jared, using the new StreamingFileSink is indeed not supported for S3 (PrestoFS or not) right now. Work on this is tracked under https://issues.apache.org/jira/browse/FLINK-9752 and should hopefully make it into the next Flink release. Bes

Re: Filescheme GS not found sometimes - inconsistent exceptions for reading from GCS

Hi Nico, Unfortunately I can't share any of data, but it is not even data being processed at the point of failure - it is still in the matching-files-from-GCS phase. I am using Apache Beam's FileIO to match files and during one of those match-files steps I get the failure above. Currently I run

Re: Filescheme GS not found sometimes - inconsistent exceptions for reading from GCS

Hi Encho, the SpillingAdaptiveSpanningRecordDeserializer that you see in your stack trace is executed while reading input records from another task. If the (serialized) records are too large (> 5MiB), it will write and assemble them in a spilling channel, i.e. on disk, instead of using memory. This

StreamingFileSink with PrestoFS?

I'm encountering an error on init with StreamingFileSink and presto-fs; before I continue down what appears to be a classpath issue, can someone stop me if StreamingFileSink doesn't support presto-fs? Error I'm seeing is: java.lang.UnsupportedOperationException: Not implemented by the PrestoS3Fil

Re: Weird behaviour after change sources in a job.

Hi, The problem is that Flink SQL does not expose the UIDs of the generated operators. We've met that issue before, but it is still not fully clear what would be the best way to this accessible. Best, Fabian 2018-09-13 5:15 GMT-04:00 Dawid Wysakowicz : > Hi Oleksandr, > > The mapping of state t

Re: Orc Sink Table

Hi Timo, David Thx for your quick answer BR Jose On Thu, 13 Sep 2018 at 12:41, Timo Walther wrote: > Hi Jose, > > you have to add additional Maven modules depending on the connector/format > you want to use. See this page [1] for more information. > > Feel free to ask further questions if the

Re: Orc Sink Table

Hi Jose, you have to add additional Maven modules depending on the connector/format you want to use. See this page [1] for more information. Feel free to ask further questions if the description is not enough for you. Regards, Timo [1] https://ci.apache.org/projects/flink/flink-docs-release

Re: Orc Sink Table

Hi Jose, Flink provides only OrcTableSource that is BatchTableSource only as of now. You can read more here[1]. As of now there is no support for orc sink file. The community is working extensively though on extending the set of supported connectors for Table API/SQL, so it might change in the fut

Re: Question regarding Streaming Resources

On 2018/09/13 03:30:28, Ken Krugler wrote: > Hi Bhaskar, > > > On 2018/09/12 20:42:22, Ken Krugler wrote: > >> Hi Bhaskar, > >> > >> I assume you don’t have 1000 streams, but rather one (keyed) stream with > >> 1000 different key values, yes? > >> > >> If so, then this one stream is phys

Orc Sink Table

Hi I am checking the documentation https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/common.html#register-a-tablesink Register a TableSink A registered TableSink can be used t

Re: Weird behaviour after change sources in a job.

Hi Oleksandr, The mapping of state to operator is done based on operator id, not on its name. That's why changing source's name might not be enough. That actually might be a valuable addition to check if the restored partitions still match with the provided topic/topic pattern. Would you like to

Re: How to add flink table jar to classpath via bin/flink

Hi Jeff, Right now the suggested procedure is to copy the flink-table dependency from /opt folder to /lib folder. You can read about it here[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/#setup On 13/09/18 10:44, Jeff Zhang wrote: > Because flink-table is a provid

Re: Weird behaviour after change sources in a job.

Hello Dawid, Thank you for the answer. In our case we did change the name of the Kafka source so we expected it shouldn’t restore state for a given Kafka source operator. Anyway shouldn’t FlinkKafkaConsumerBase have a safeguard which do not allow restoring of the KafkaTopicPartitions from the

Re: Broadcast managed state

Hi Deepya, It depends how do you want to update this state. If your use-case follows the Broadcast State Pattern[1], that means you perform updates to your state only based on one broadcasted stream (often called control stream) you should check out the linked docs. If you want to perform updates

Re: Logging metrics from within Elasticsearch ActionRequestFailureHandler

Thank you Dawid. I will open a Jira Wish for this. Regarding a solution for this, I have another question regarding the current implementation of ElasticsearchSink classes. Why do we use the Builder mechanism? Would that make it more inconvenient to solve issues like the one I'm having? Thanks an

How to add flink table jar to classpath via bin/flink

Because flink-table is a provided dependency, so it won't be included in the final shaded jar. I didn't find way to add custom jar to classpath via bin/flink, does anyone know that ? Thanks

Re: Logging metrics from within Elasticsearch ActionRequestFailureHandler

Hi Averell, I am afraid this is currently not possible without modifying ElasticsearchSink. Would you like to open jira ticket for metrics access in the failureHandler? Best, Dawid On 13/09/18 03:50, Averell wrote: > Good day everyone, > > I'm writing to Elasticsearch, and I need to count the

Re: How to get taskmanager hostname and port on runtime

Hi, For the hostname, by default identifier for metrics scoped to taskmanager contain its hostname. You can read more about metrics scope here[1]. What do you mean by port? Which port would you like to report? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.ht

Re: Weird behaviour after change sources in a job.

Hi Juan, I think this is somehow expected behaviour. Flink, in order to provide proper processing semantics keeps track of partitions offsets internally, and checkpoints those offsets. FlinkKafkaConsumer supports also new partitions discovery. Having in mind both of those features, if you restart

How to handle exceptions in Kafka sink?

I have a Flink job that writes data into Kafka. The Kafka topic has maximum message size set to 5 MB, so if I try to write any record larger than 5 MB, it throws the following exception and brings the job down.