[jira] [Created] (FLINK-14807) Add Table#collect api for fetching data to client

2019-11-14 Thread Jeff Zhang (Jira)
Jeff Zhang created FLINK-14807: -- Summary: Add Table#collect api for fetching data to client Key: FLINK-14807 URL: https://issues.apache.org/jira/browse/FLINK-14807 Project: Flink Issue Type:

[jira] [Created] (FLINK-14806) Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter

2019-11-14 Thread Zhenghua Gao (Jira)
Zhenghua Gao created FLINK-14806: Summary: Introduce setTimestamp/getTimestamp interface to TypeGetterSetters/VectorizedColumnBatch and writeTimestamp interface to BinaryWriter Key: FLINK-14806 URL:

[jira] [Created] (FLINK-14805) Remove unixDateCeil related code

2019-11-14 Thread Zhenghua Gao (Jira)
Zhenghua Gao created FLINK-14805: Summary: Remove unixDateCeil related code Key: FLINK-14805 URL: https://issues.apache.org/jira/browse/FLINK-14805 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14804) The flink that starts stand-alone mode appears abnormal

2019-11-14 Thread FengXiaohu (Jira)
FengXiaohu created FLINK-14804: -- Summary: The flink that starts stand-alone mode appears abnormal Key: FLINK-14804 URL: https://issues.apache.org/jira/browse/FLINK-14804 Project: Flink Issue

[jira] [Created] (FLINK-14803) Support Consistency Level for InfluxDB metrics reporter

2019-11-14 Thread ouyangwulin (Jira)
ouyangwulin created FLINK-14803: --- Summary: Support Consistency Level for InfluxDB metrics reporter Key: FLINK-14803 URL: https://issues.apache.org/jira/browse/FLINK-14803 Project: Flink Issue

[jira] [Created] (FLINK-14802) Multi vectorized read version support for hive orc read

2019-11-14 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14802: Summary: Multi vectorized read version support for hive orc read Key: FLINK-14802 URL: https://issues.apache.org/jira/browse/FLINK-14802 Project: Flink

[jira] [Created] (FLINK-14801) flink sql parser unparser func partition frame local variable name mistake

2019-11-14 Thread xiaodao (Jira)
xiaodao created FLINK-14801: --- Summary: flink sql parser unparser func partition frame local variable name mistake Key: FLINK-14801 URL: https://issues.apache.org/jira/browse/FLINK-14801 Project: Flink

[jira] [Created] (FLINK-14800) Introduce parallelism inference for HiveSource

2019-11-14 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14800: Summary: Introduce parallelism inference for HiveSource Key: FLINK-14800 URL: https://issues.apache.org/jira/browse/FLINK-14800 Project: Flink Issue Type:

[jira] [Created] (FLINK-14799) Improve batch sql and hive integrate performance milestone-2

2019-11-14 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14799: Summary: Improve batch sql and hive integrate performance milestone-2 Key: FLINK-14799 URL: https://issues.apache.org/jira/browse/FLINK-14799 Project: Flink

Re: [DISCUSS] Releasing Flink 1.8.3

2019-11-14 Thread Hequn Cheng
Hi Chesnay, Great to hear that jackson-2.10.1 works well on master. Really a good job! - Whether backport this change to 1.8/1.9 I had taken a quick look at the security vulnerabilities, some of them seem can lead to high-security problems, thus from my point of view, I'm in favor of adding the

[jira] [Created] (FLINK-14798) how to setup a ha flink cluster on k8s?

2019-11-14 Thread rock (Jira)
rock created FLINK-14798: Summary: how to setup a ha flink cluster on k8s? Key: FLINK-14798 URL: https://issues.apache.org/jira/browse/FLINK-14798 Project: Flink Issue Type: New Feature

Re: flink checkpoint state data corruption

2019-11-14 Thread Yun Tang
Hi Jeffrey Thanks for your willingness to try this un-merged PR. Would you please share your exception stack and your proto file? If you could just summed up a job which could be able to reproduce this bug and share this to me, that would be better to fix this problem. Best Yun Tang From:

Re: [DISCUSS] Expose or setup a secur...@flink.apache.org mailing list for security report and discussion

2019-11-14 Thread Becket Qin
Thanks for bringing this up, Dian. +1 on creating a project specific security mailing list. My two cents, I think it is worth doing in practice. Although the ASF security ML is always available, usually all the emails are simply routed to the individual project PMC. This is an additional hop.

[jira] [Created] (FLINK-14797) Remove power mockito from RemoteStreamExecutionEnvironmentTest

2019-11-14 Thread Zili Chen (Jira)
Zili Chen created FLINK-14797: - Summary: Remove power mockito from RemoteStreamExecutionEnvironmentTest Key: FLINK-14797 URL: https://issues.apache.org/jira/browse/FLINK-14797 Project: Flink

[jira] [Created] (FLINK-14796) Add document about limitations of different Hive versions

2019-11-14 Thread Rui Li (Jira)
Rui Li created FLINK-14796: -- Summary: Add document about limitations of different Hive versions Key: FLINK-14796 URL: https://issues.apache.org/jira/browse/FLINK-14796 Project: Flink Issue Type:

Re: flink checkpoint state data corruption

2019-11-14 Thread Yu Li
@Yun Tang As the author of the referenced PR, it would be great if you could help take a look here. Thanks. @Jeffery: For your second question, with officially released Flink, once checkpoint has been completed successfully it's safe to restore from [1]. The issue you encountered probably was

Re: [DISCUSS] Expose or setup a secur...@flink.apache.org mailing list for security report and discussion

2019-11-14 Thread Yu Li
Thanks for bringing up this discussion Dian! How to report security bugs to our project is a very important topic! Big +1 on adding some explicit instructions in our document about how to report security issues, and I suggest to open another thread to vote the reporting way in Flink. FWIW, known

[jira] [Created] (FLINK-14795) Rework TaskExecutorPartitionLifecycleTest

2019-11-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-14795: Summary: Rework TaskExecutorPartitionLifecycleTest Key: FLINK-14795 URL: https://issues.apache.org/jira/browse/FLINK-14795 Project: Flink Issue

[jira] [Created] (FLINK-14794) KeyedStream#transform not working with factories

2019-11-14 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-14794: --- Summary: KeyedStream#transform not working with factories Key: FLINK-14794 URL: https://issues.apache.org/jira/browse/FLINK-14794 Project: Flink Issue Type:

[jira] [Created] (FLINK-14793) Add SPI interface to discovery state backend and establish well-known names

2019-11-14 Thread Kezhu Wang (Jira)
Kezhu Wang created FLINK-14793: -- Summary: Add SPI interface to discovery state backend and establish well-known names Key: FLINK-14793 URL: https://issues.apache.org/jira/browse/FLINK-14793 Project:

[jira] [Created] (FLINK-14792) Extend TaskExecutor interface to support release of cluster partitions

2019-11-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-14792: Summary: Extend TaskExecutor interface to support release of cluster partitions Key: FLINK-14792 URL: https://issues.apache.org/jira/browse/FLINK-14792

[jira] [Created] (FLINK-14791) Process ClusterPartitionReport on ResourceManager

2019-11-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-14791: Summary: Process ClusterPartitionReport on ResourceManager Key: FLINK-14791 URL: https://issues.apache.org/jira/browse/FLINK-14791 Project: Flink

Re: [DISCUSS] Flink Avro Cloudera Registry (FLINK-14577)

2019-11-14 Thread Őrhidi Mátyás
Thanks Dawid, Gyula we will create a pull request then with the proposed changes were we can further elaborate on the dependencies. Regards, Matyas On Thu, Nov 14, 2019 at 11:10 AM Dawid Wysakowicz wrote: > Hi Gyula, > > First of all sorry for the delayed response. > > I see the argument for

[jira] [Created] (FLINK-14790) Custom state backend in job graph composition without depending on concrete state backend implementation

2019-11-14 Thread Kezhu Wang (Jira)
Kezhu Wang created FLINK-14790: -- Summary: Custom state backend in job graph composition without depending on concrete state backend implementation Key: FLINK-14790 URL:

[jira] [Created] (FLINK-14789) extends max/min type in ColumnStats from Number to Comparable

2019-11-14 Thread godfrey he (Jira)
godfrey he created FLINK-14789: -- Summary: extends max/min type in ColumnStats from Number to Comparable Key: FLINK-14789 URL: https://issues.apache.org/jira/browse/FLINK-14789 Project: Flink

Re: [DISCUSS] FLIP-86: Improve Connector Properties

2019-11-14 Thread Kurt Young
+1 to the general idea of this FLIP. Regarding to connector properties, IIUC it would be divided into 2 parts: 1. connector.key1 = value1, these are interpreted by Flink framework and do whatever needed for such property. 2. connector.properties.xxx.key = value. There items are prefixed with

[jira] [Created] (FLINK-14788) Add configure method to CheckpointConfig

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14788: Summary: Add configure method to CheckpointConfig Key: FLINK-14788 URL: https://issues.apache.org/jira/browse/FLINK-14788 Project: Flink Issue Type:

[jira] [Created] (FLINK-14787) Add configure method to StreamExecutionEnvironment

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14787: Summary: Add configure method to StreamExecutionEnvironment Key: FLINK-14787 URL: https://issues.apache.org/jira/browse/FLINK-14787 Project: Flink

[jira] [Created] (FLINK-14785) Enable execution configuration from Configuration object

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14785: Summary: Enable execution configuration from Configuration object Key: FLINK-14785 URL: https://issues.apache.org/jira/browse/FLINK-14785 Project: Flink

[jira] [Created] (FLINK-14786) Add configure method to ExecutionConfig

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14786: Summary: Add configure method to ExecutionConfig Key: FLINK-14786 URL: https://issues.apache.org/jira/browse/FLINK-14786 Project: Flink Issue Type:

[RESULT][VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-14 Thread Dawid Wysakowicz
Thank you all for your votes. There were 7 +1 votes, 4 of which are binding: - Aljoscha (binding) - Tison (binding) - Kostas (binding) - Gyula (binding) - Danny (non-binding) - Jeff (non-binding) - Jingsong (non-binding) There were no disapproving votes. FLIP-59 has been accepted. Best,

[jira] [Created] (FLINK-14784) CsvTableSink miss delimiter when row start with null member

2019-11-14 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-14784: -- Summary: CsvTableSink miss delimiter when row start with null member Key: FLINK-14784 URL: https://issues.apache.org/jira/browse/FLINK-14784 Project: Flink

[jira] [Created] (FLINK-14783) Implement a benchmark for `ContinuousFileReaderOperator`

2019-11-14 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-14783: - Summary: Implement a benchmark for `ContinuousFileReaderOperator` Key: FLINK-14783 URL: https://issues.apache.org/jira/browse/FLINK-14783 Project: Flink

[jira] [Created] (FLINK-14782) CoreModule#getFunctionDefinition throws exception when function does not exist

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14782: Summary: CoreModule#getFunctionDefinition throws exception when function does not exist Key: FLINK-14782 URL: https://issues.apache.org/jira/browse/FLINK-14782

[jira] [Created] (FLINK-14781) clarify that a RocksDB dependency in pom.xml may not be needed

2019-11-14 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-14781: --- Summary: clarify that a RocksDB dependency in pom.xml may not be needed Key: FLINK-14781 URL: https://issues.apache.org/jira/browse/FLINK-14781 Project: Flink

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-14 Thread Jingsong Li
+1 (non-binding) Thanks Dawid for driving this. Best, Jingsong Lee On Thu, Nov 14, 2019 at 6:09 PM Gyula Fóra wrote: > +1 (binding) > > On Thu, Nov 14, 2019 at 11:07 AM Jeff Zhang wrote: > > > +1 (non-binding) > > > > Kostas Kloudas 于2019年11月14日周四 下午6:04写道: > > > > > +1 (binding) > > > > > >

Re: [DISCUSS] Expose or setup a secur...@flink.apache.org mailing list for security report and discussion

2019-11-14 Thread Chesnay Schepler
Source: https://www.apache.org/security/ Now, we can of course setup such a mailing list (as outlined here https://www.apache.org/security/committers.html), but I'm not sure if it is necessary since the number of reports is _really_ low. On 14/11/2019 11:03, Chesnay Schepler wrote: AFAIK,

Re: [DISCUSS] Flink Avro Cloudera Registry (FLINK-14577)

2019-11-14 Thread Dawid Wysakowicz
Hi Gyula, First of all sorry for the delayed response. I see the argument for handling metadata from kafka headers. I haven't noticed the schema you are proposing is actually KafkaDeserializationSchema, which means it works only with Kafka. I still believe it would be really beneficial for the

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-14 Thread Gyula Fóra
+1 (binding) On Thu, Nov 14, 2019 at 11:07 AM Jeff Zhang wrote: > +1 (non-binding) > > Kostas Kloudas 于2019年11月14日周四 下午6:04写道: > > > +1 (binding) > > > > On Tue, Nov 12, 2019 at 10:20 AM tison wrote: > > > > > > +1 (binding) > > > > > > Best, > > > tison. > > > > > > > > > Aljoscha Krettek

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-14 Thread Jeff Zhang
+1 (non-binding) Kostas Kloudas 于2019年11月14日周四 下午6:04写道: > +1 (binding) > > On Tue, Nov 12, 2019 at 10:20 AM tison wrote: > > > > +1 (binding) > > > > Best, > > tison. > > > > > > Aljoscha Krettek 于2019年11月12日周二 下午4:37写道: > > > > > +1 (binding) > > > > > > > On 8. Nov 2019, at 09:45, Danny

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-14 Thread Kostas Kloudas
+1 (binding) On Tue, Nov 12, 2019 at 10:20 AM tison wrote: > > +1 (binding) > > Best, > tison. > > > Aljoscha Krettek 于2019年11月12日周二 下午4:37写道: > > > +1 (binding) > > > > > On 8. Nov 2019, at 09:45, Danny Chan wrote: > > > > > > Thanks for driving on this, David. > > > > > > +1(non-binding)

Re: [DISCUSS] Expose or setup a secur...@flink.apache.org mailing list for security report and discussion

2019-11-14 Thread Chesnay Schepler
AFAIK, the official way to report vulnerabilities in any apache project is to write to secur...@apache.org and/or notify the respective PMC. So far, we had several reports that went this route, hence I'm not convinced that an additional ML is required. I would be fine with an additional

Re: [DISCUSS] Releasing Flink 1.8.3

2019-11-14 Thread Chesnay Schepler
So here's the state of things: The master of flink-shaded now uses jackson 2.10.1, which eliminates a whole category of security vulnerabilities. The flink master works perfectly fine with that version; 1.9 will likely do so too and 1.8 would require a minor adjustment. Hence, there may be

[jira] [Created] (FLINK-14780) Avoid leaking instance of DefaultScheduler before object is constructed

2019-11-14 Thread Gary Yao (Jira)
Gary Yao created FLINK-14780: Summary: Avoid leaking instance of DefaultScheduler before object is constructed Key: FLINK-14780 URL: https://issues.apache.org/jira/browse/FLINK-14780 Project: Flink

Re: [DISCUSS] FLIP-86: Improve Connector Properties

2019-11-14 Thread Dawid Wysakowicz
Yes, personally I would also go with the external system's format for the exact reason you mentioned. Best, Dawid On 14/11/2019 10:32, Jark Wu wrote: > Hi Dawid, > > The `connector.properties.zookeeper.connect` sounds good to me. > > Just to clarify, in this way, we can support arbitrary

Re: [DISCUSS] FLIP-86: Improve Connector Properties

2019-11-14 Thread Jark Wu
Hi Dawid, The `connector.properties.zookeeper.connect` sounds good to me. Just to clarify, in this way, we can support arbitrary properties and we can document the important ones. And regarding to the value format, shall we use the external system's format or use Flink's format, e.g. the list

[jira] [Created] (FLINK-14779) Migrate duration and memory size ConfigOptions in PythonOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14779: Summary: Migrate duration and memory size ConfigOptions in PythonOptions Key: FLINK-14779 URL: https://issues.apache.org/jira/browse/FLINK-14779 Project:

[jira] [Created] (FLINK-14778) Migrate duration and memory size ConfigOptions in HistoryServerOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14778: Summary: Migrate duration and memory size ConfigOptions in HistoryServerOptions Key: FLINK-14778 URL: https://issues.apache.org/jira/browse/FLINK-14778

Re: [DISCUSS] FLIP-86: Improve Connector Properties

2019-11-14 Thread Dawid Wysakowicz
Hi Jark, I see the not readable point and that is also my concern about my suggestion with all the properties under a single key as the list might get quite long. What do you think about modifying your suggestion a bit and at least add the `properties` prefix? So that we can distinguish the

[jira] [Created] (FLINK-14777) Migrate duration and memory size ConfigOptions in MetricOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14777: Summary: Migrate duration and memory size ConfigOptions in MetricOptions Key: FLINK-14777 URL: https://issues.apache.org/jira/browse/FLINK-14777 Project:

[jira] [Created] (FLINK-14776) Migrate duration and memory size ConfigOptions in RocksDBConfigurableOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14776: Summary: Migrate duration and memory size ConfigOptions in RocksDBConfigurableOptions Key: FLINK-14776 URL: https://issues.apache.org/jira/browse/FLINK-14776

[jira] [Created] (FLINK-14775) Migrate duration and memory size ConfigOptions in CheckpointingOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14775: Summary: Migrate duration and memory size ConfigOptions in CheckpointingOptions Key: FLINK-14775 URL: https://issues.apache.org/jira/browse/FLINK-14775

[jira] [Created] (FLINK-14774) Migrate duration and memory size ConfigOptions in HighAvailabilityOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14774: Summary: Migrate duration and memory size ConfigOptions in HighAvailabilityOptions Key: FLINK-14774 URL: https://issues.apache.org/jira/browse/FLINK-14774

[jira] [Created] (FLINK-14773) Migrate duration and memory size ConfigOptions in MesosOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14773: Summary: Migrate duration and memory size ConfigOptions in MesosOptions Key: FLINK-14773 URL: https://issues.apache.org/jira/browse/FLINK-14773 Project:

[jira] [Created] (FLINK-14772) Migrate duration and memory size ConfigOptions in YarnConfigOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14772: Summary: Migrate duration and memory size ConfigOptions in YarnConfigOptions Key: FLINK-14772 URL: https://issues.apache.org/jira/browse/FLINK-14772 Project:

[jira] [Created] (FLINK-14771) Migrate duration and memory size ConfigOptions in ResourceManagerOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14771: Summary: Migrate duration and memory size ConfigOptions in ResourceManagerOptions Key: FLINK-14771 URL: https://issues.apache.org/jira/browse/FLINK-14771

[jira] [Created] (FLINK-14770) Migrate duration and memory size ConfigOptions in WebOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14770: Summary: Migrate duration and memory size ConfigOptions in WebOptions Key: FLINK-14770 URL: https://issues.apache.org/jira/browse/FLINK-14770 Project: Flink

[jira] [Created] (FLINK-14769) Migrate duration and memory size ConfigOptions in NettyShuffleEnvironmentOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14769: Summary: Migrate duration and memory size ConfigOptions in NettyShuffleEnvironmentOptions Key: FLINK-14769 URL: https://issues.apache.org/jira/browse/FLINK-14769

[jira] [Created] (FLINK-14768) Migrate duration and memory size ConfigOptions in SecurityOptions

2019-11-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14768: Summary: Migrate duration and memory size ConfigOptions in SecurityOptions Key: FLINK-14768 URL: https://issues.apache.org/jira/browse/FLINK-14768 Project: