[jira] [Created] (FLINK-20015) Failure to execute streaming query

2020-11-05 Thread Satyam Shekhar (Jira)
Satyam Shekhar created FLINK-20015: -- Summary: Failure to execute streaming query Key: FLINK-20015 URL: https://issues.apache.org/jira/browse/FLINK-20015 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-20016) Support TimestampAssigner and WatermarkGenerator for Python DataStream API.

2020-11-05 Thread Shuiqiang Chen (Jira)
Shuiqiang Chen created FLINK-20016: -- Summary: Support TimestampAssigner and WatermarkGenerator for Python DataStream API. Key: FLINK-20016 URL: https://issues.apache.org/jira/browse/FLINK-20016

[jira] [Created] (FLINK-20014) Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler

2020-11-05 Thread Alan Leong (Jira)
Alan Leong created FLINK-20014: -- Summary: Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler Key: FLINK-20014 URL: https://issues.apache.org/jira/browse/FLINK-20014 Project: Flink

[jira] [Created] (FLINK-20013) BoundedBlockingSubpartition may leak network buffer if task is failed or canceled

2020-11-05 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-20013: --- Summary: BoundedBlockingSubpartition may leak network buffer if task is failed or canceled Key: FLINK-20013 URL: https://issues.apache.org/jira/browse/FLINK-20013

Re: Investigating flinK

2020-11-05 Thread Kaibo Zhou
Hi, you can write a custom log appender that modifies the logs before they are sent. Thanks. Diana El-Masri 于2020年11月6日周五 上午7:47写道: > Hi, > > No the logs of the sources connected to flink. > > Thanks > > Chesnay Schepler a écrit : > > > Are you referring to the log files of Flink? > > > > On

Re: [DISCUSS] Move license check utility to a new repository to share it with flink-statefun

2020-11-05 Thread Tzu-Li (Gordon) Tai
Hi Robert, I think this could be useful in flink-statefun. StateFun currently has two modules that bundles dependencies, most importantly the `flink-statefun-distribution` module which currently bundles some Flink dependencies as well as Flink connectors (Kafka, Kinesis). Upgrading the Flink

[jira] [Created] (FLINK-20012) Hive 3.1 integration exception

2020-11-05 Thread Dino Zhang (Jira)
Dino Zhang created FLINK-20012: -- Summary: Hive 3.1 integration exception Key: FLINK-20012 URL: https://issues.apache.org/jira/browse/FLINK-20012 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-20011) PageRankITCase.testPrintWithRMatGraph hangs

2020-11-05 Thread Dian Fu (Jira)
Dian Fu created FLINK-20011: --- Summary: PageRankITCase.testPrintWithRMatGraph hangs Key: FLINK-20011 URL: https://issues.apache.org/jira/browse/FLINK-20011 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-20010) SinkITCase.writerAndCommitterAndGlobalCommitterExecuteInStreamingMode fails on Azure Pipeline

2020-11-05 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-20010: --- Summary: SinkITCase.writerAndCommitterAndGlobalCommitterExecuteInStreamingMode fails on Azure Pipeline Key: FLINK-20010 URL: https://issues.apache.org/jira/browse/FLINK-20010

Re: Investigating flinK

2020-11-05 Thread Diana El-Masri
Hi, No the logs of the sources connected to flink. Thanks Chesnay Schepler a écrit : Are you referring to the log files of Flink? On 11/5/2020 7:01 PM, Diana El-Masri wrote: Hi, I am starting my PhD at "Ecole Polytechnique of Montreal" on IoT log management. I am considering using

Re: Investigating flinK

2020-11-05 Thread Chesnay Schepler
Are you referring to the log files of Flink? On 11/5/2020 7:01 PM, Diana El-Masri wrote: Hi, I am starting my PhD at "Ecole Polytechnique of Montreal" on IoT log management. I am considering using Flink for my edge layer processing. Could please advise if there is a possibility to write a

[jira] [Created] (FLINK-20009) Add 404 check to docs build

2020-11-05 Thread Seth Wiesman (Jira)
Seth Wiesman created FLINK-20009: Summary: Add 404 check to docs build Key: FLINK-20009 URL: https://issues.apache.org/jira/browse/FLINK-20009 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-20008) Java Deadlock in ZooKeeperLeaderElectionTest.testZooKeeperReelectionWithReplacement()

2020-11-05 Thread Stephan Ewen (Jira)
Stephan Ewen created FLINK-20008: Summary: Java Deadlock in ZooKeeperLeaderElectionTest.testZooKeeperReelectionWithReplacement() Key: FLINK-20008 URL: https://issues.apache.org/jira/browse/FLINK-20008

Investigating flinK

2020-11-05 Thread Diana El-Masri
Hi, I am starting my PhD at "Ecole Polytechnique of Montreal" on IoT log management. I am considering using Flink for my edge layer processing. Could please advise if there is a possibility to write a flink plugin that intercepts and modify the logs before they are sent to the

[jira] [Created] (FLINK-20007) SinkTransformationTranslator fail to handle the PartitionTransformation

2020-11-05 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20007: - Summary: SinkTransformationTranslator fail to handle the PartitionTransformation Key: FLINK-20007 URL: https://issues.apache.org/jira/browse/FLINK-20007 Project: Flink

[jira] [Created] (FLINK-20006) FileSinkITCase.testFileSink: The record 0 should occur 4 times, but only occurs 8time expected:<4> but was:<8>

2020-11-05 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20006: -- Summary: FileSinkITCase.testFileSink: The record 0 should occur 4 times, but only occurs 8time expected:<4> but was:<8> Key: FLINK-20006 URL:

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread DEEP NARAYAN Singh
Thanks Prasanna & Till for quick response. Looks like my use case is very similar to yours ,I will try to run multiple containers on the same machine and will update you accordingly. Thanks , -Deep On Thu, Nov 5, 2020 at 2:33 PM Till Rohrmann wrote: > Hi Deep, > > you can increase the

[jira] [Created] (FLINK-20005) "Kerberized YARN application" test unstable

2020-11-05 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20005: -- Summary: "Kerberized YARN application" test unstable Key: FLINK-20005 URL: https://issues.apache.org/jira/browse/FLINK-20005 Project: Flink Issue Type:

[jira] [Created] (FLINK-20004) UpperLimitExceptionParameter description is misleading

2020-11-05 Thread Flavio Pompermaier (Jira)
Flavio Pompermaier created FLINK-20004: -- Summary: UpperLimitExceptionParameter description is misleading Key: FLINK-20004 URL: https://issues.apache.org/jira/browse/FLINK-20004 Project: Flink

[DISCUSS] Move license check utility to a new repository to share it with flink-statefun

2020-11-05 Thread Robert Metzger
Hi all, for the upcoming 1.12 release (and beyond ;) ), I added a utility [1] that checks for the most common licensing issues. It doesn't find everything, but the by far most common case of forgetting to add a version upgrade or changed transitive dependency to the NOTICE file is covered.

[jira] [Created] (FLINK-20003) Improve slot report logging messages

2020-11-05 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-20003: Summary: Improve slot report logging messages Key: FLINK-20003 URL: https://issues.apache.org/jira/browse/FLINK-20003 Project: Flink Issue Type:

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-11-05 Thread Yun Tang
For your information, I executed the flink-benchmarks [1] twice within docker container based on flink-1.11.1 scala_2.11-java8-debian image [2], from the results of state benchmarks, I cannot see obvious performance changes whether to use jemalloc as memory allocator: RocksDB keyed state

[jira] [Created] (FLINK-20002) Add a StreamExecutionEnvironment#getExecutionEnvironment(Configuration) method

2020-11-05 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-20002: Summary: Add a StreamExecutionEnvironment#getExecutionEnvironment(Configuration) method Key: FLINK-20002 URL: https://issues.apache.org/jira/browse/FLINK-20002

[jira] [Created] (FLINK-20001) Don't use setAllVerticesInSameSlotSharingGroupByDefault in StreamGraphGenerator

2020-11-05 Thread Aljoscha Krettek (Jira)
Aljoscha Krettek created FLINK-20001: Summary: Don't use setAllVerticesInSameSlotSharingGroupByDefault in StreamGraphGenerator Key: FLINK-20001 URL: https://issues.apache.org/jira/browse/FLINK-20001

[jira] [Created] (FLINK-20000) Extend the license checker to ensure that 3rd party licenses are shipped

2020-11-05 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-2: -- Summary: Extend the license checker to ensure that 3rd party licenses are shipped Key: FLINK-2 URL: https://issues.apache.org/jira/browse/FLINK-2

[jira] [Created] (FLINK-19999) State Processor API classes leaking into savepoint

2020-11-05 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-1: --- Summary: State Processor API classes leaking into savepoint Key: FLINK-1 URL: https://issues.apache.org/jira/browse/FLINK-1 Project: Flink Issue Type:

[jira] [Created] (FLINK-19998) Invalid Link to checkpoints and Savepoints in stateful stream processing concepts

2020-11-05 Thread Aditya Agarwal (Jira)
Aditya Agarwal created FLINK-19998: -- Summary: Invalid Link to checkpoints and Savepoints in stateful stream processing concepts Key: FLINK-19998 URL: https://issues.apache.org/jira/browse/FLINK-19998

[jira] [Created] (FLINK-19997) Implement an e2e test for sql-client with Confluent Registry Avro format

2020-11-05 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-19997: Summary: Implement an e2e test for sql-client with Confluent Registry Avro format Key: FLINK-19997 URL: https://issues.apache.org/jira/browse/FLINK-19997

[jira] [Created] (FLINK-19996) Add end-to-end IT case for Debezium + Kafka + temporal join

2020-11-05 Thread Jark Wu (Jira)
Jark Wu created FLINK-19996: --- Summary: Add end-to-end IT case for Debezium + Kafka + temporal join Key: FLINK-19996 URL: https://issues.apache.org/jira/browse/FLINK-19996 Project: Flink Issue

[jira] [Created] (FLINK-19995) 【Flink SQL Client】Use Flink Kafka Connector has more one exception

2020-11-05 Thread zhisheng (Jira)
zhisheng created FLINK-19995: Summary: 【Flink SQL Client】Use Flink Kafka Connector has more one exception Key: FLINK-19995 URL: https://issues.apache.org/jira/browse/FLINK-19995 Project: Flink

[jira] [Created] (FLINK-19994) All vertices in an DataSet iteration job will be eagerly scheduled

2020-11-05 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-19994: --- Summary: All vertices in an DataSet iteration job will be eagerly scheduled Key: FLINK-19994 URL: https://issues.apache.org/jira/browse/FLINK-19994 Project: Flink

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread Till Rohrmann
Hi Deep, you can increase the average CPU load by reducing the number of overall resources. Having fewer slots over which you can distribute the work should increase the resource usage. Cheers, Till On Thu, Nov 5, 2020 at 9:03 AM Prasanna kumar wrote: > Deep, > > 1) Is it a cpu/memory/io

[jira] [Created] (FLINK-19993) Remove the flink-connector-filesystem module

2020-11-05 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19993: -- Summary: Remove the flink-connector-filesystem module Key: FLINK-19993 URL: https://issues.apache.org/jira/browse/FLINK-19993 Project: Flink Issue Type:

[jira] [Created] (FLINK-19992) Integrate new orc to Hive source

2020-11-05 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19992: Summary: Integrate new orc to Hive source Key: FLINK-19992 URL: https://issues.apache.org/jira/browse/FLINK-19992 Project: Flink Issue Type: Sub-task

Re: [VOTE] Remove flink-connector-filesystem module.

2020-11-05 Thread Kostas Kloudas
Hi all, Given the current outcome of the voting process, the proposal for removing the "flink-connector-filesystem module" has passed. I will move on with opening a JIRA and actually removing the module and all the related code. Thank you all for voting, Kostas On Fri, Oct 30, 2020 at 5:05 PM

[jira] [Created] (FLINK-19991) UnalignedCheckpointITCase#shouldPerformUnalignedCheckpointMassivelyParallel fails on Azure Pipeline

2020-11-05 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-19991: --- Summary: UnalignedCheckpointITCase#shouldPerformUnalignedCheckpointMassivelyParallel fails on Azure Pipeline Key: FLINK-19991 URL:

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread Prasanna kumar
Deep, 1) Is it a cpu/memory/io intensive job ?? Based on that you could allocate resources. >From the question, if the CPU is not utilised , you could run multiple containers on the same machine(tm) ... Following may not be exact case as yours but to give you an idea. Few months back I have