Re: Stopping a job

Hi Arvid:    Thanks for the links.   A few questions: 1. Is there any particular interface in 1.9+ that identifies the source as stoppable ?2. Is there any distinction b/w stop and cancel  in 1.9+ ?3. Is there any list of sources which are documented as stoppable besides the one listed in

Flink not restoring from checkpoint when job manager fails even with HA

Hi, We are running Flink in K8S. We used https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/jobmanager_high_availability.html to set high availability. We set max number of retries for a task to 2. After task fails twice and then the job manager fails. This is expected. But

Failed to deserialize Avro record

We are using AvroRowDeserializationSchema with Kafka Table source to deserialize the messages. Application failed with "Failed to deserialize Avro record." for different messages it seems. Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -26 Caused by:

Re: Stopping a job

Yes, it seems as if FlinkKinesisConsumer does not implement it. Here are the links to the respective javadoc [1] and code [2]. Note that in later releases (1.9+) this interface has been removed. Stop is now implemented through a cancel() on source level. In general, I don't think that in a

Re: Stopping a job

Hi Arvid: I check the link and it indicates that only Storm SpoutSource, TwitterSource and NifiSource support stop.    Does this mean that FlinkKinesisConsumer is not stoppable ? Also, can you please point me to the Stoppable interface mentioned in the link ?  I found the following but am not

??????flink 1.9 ????????????????

Hi,star KafkaConnectorupsert[1] [1]https://mp.weixin.qq.com/s/MSs7HSaegyWWU3Fig2PYYA | | Sun.Zhu | | 17626017...@163.com | ?? ??2020??06??3?? 14:47??star<3149768...@qq.com> ??

Re: Flink s3 streaming performance

Thanks Arvid! Will try to increase the property you recommended and will post the update. On Sat, Jun 6, 2020, 7:33 AM Arvid Heise wrote: > Hi Venkata, > > you can find them on the Hadoop AWS page (we are just using it as a > library) [1]. > > [1] >

Re: Run command after Batch is finished

It would run in the client side where ExecutionEnvironment is created. Mark Davis 于2020年6月6日周六 下午8:14写道： > Hi Jeff, > > Thank you very much! That is exactly what I need. > > Where the listener code will run in the cluster deployment(YARN, k8s)? > Will it be sent over the network? > > Thank you!

flink1.9集成pushgateway和prometheus版本问题

请教下大佬们，想知道flink1.9.0版本对应pushgateway和prometheus的版本号分别都是多少。

Re: Run command after Batch is finished

Hi Jeff, Thank you very much! That is exactly what I need. Where the listener code will run in the cluster deployment(YARN, k8s)? Will it be sent over the network? Thank you! Mark ‐‐‐ Original Message ‐‐‐ On Friday, June 5, 2020 6:13 PM, Jeff Zhang wrote: > You can try JobListener

Re: Flink s3 streaming performance

Hi Venkata, you can find them on the Hadoop AWS page (we are just using it as a library) [1]. [1] https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration On Sat, Jun 6, 2020 at 1:26 AM venkata sateesh` kolluru < vkollur...@gmail.com> wrote:

Flink Stream job to parquet sink

Hello All, I am receiving a set of events in Avro format on different topics. I want to consume these and write to s3 in parquet format. I have written a below job that creates a different stream for each event and fetches it schema from the confluent schema registry to create a parquet sink for

Data Quality Library in Flink

Hello All, I want to do some data quality analysis on stream data example. 1. Fill rate in a particular column 2. How many events are going to error queue due to favor schema validation failed? 3. Different statistics measure of a column. 3. Alert if a particular threshold is breached (like if