[jira] [Created] (FLINK-16627) when insert into kafka ,how can i remove the keys with null value of json

2020-03-16 Thread jackray wang (Jira)
jackray wang created FLINK-16627: Summary: when insert into kafka ,how can i remove the keys with null value of json Key: FLINK-16627 URL: https://issues.apache.org/jira/browse/FLINK-16627 Project:

[jira] [Created] (FLINK-16626) Exception Encountered when cancelling a job in yarn per-job mode

2020-03-16 Thread chaiyongqiang (Jira)
chaiyongqiang created FLINK-16626: - Summary: Exception Encountered when cancelling a job in yarn per-job mode Key: FLINK-16626 URL: https://issues.apache.org/jira/browse/FLINK-16626 Project: Flink

Re: Flink YARN Job unexpectedly switched to SUCCEEDED final status after CSV deserialization failure

2020-03-16 Thread DONG, Weike
Hi dev, After debugging and analysis, eventually the culprit is found. In order to cope with occasional data-format errors and intermittent source exceptions (Internet connection is somewhat unstable in my environment) that interrupt my Flink jobs occasionally, I have added a broad catch block

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

2020-03-16 Thread Kurt Young
Looks like I hit the gmail's bug again... Best, Kurt On Tue, Mar 17, 2020 at 11:11 AM Wei Zhong wrote: > Hi Kurt, > > This vote thread is independent from my side[1]. If this thread is > combined with another thread from your side, you can try to change the mail > client. > > Best, > Wei > >

Flink YARN Job unexpectedly switched to SUCCEEDED final status after CSV deserialization failure

2020-03-16 Thread DONG, Weike
Hi community, I have noticed that when a wrong CSV record is ingested and deserialized at CsvRowDeserializationSchema, when ignoreParseErrors is set to false, then an IOException is thrown, which is expected, and in earlier Flink versions, the Flink YARN app would terminated with FINISHED state

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

2020-03-16 Thread Wei Zhong
Hi Kurt, This vote thread is independent from my side[1]. If this thread is combined with another thread from your side, you can try to change the mail client. Best, Wei [1]

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

2020-03-16 Thread Kurt Young
Hi, please use a dedicated vote thread. Best, Kurt On Tue, Mar 17, 2020 at 10:36 AM jincheng sun wrote: > +1 > > Best, > Jincheng > > > > Wei Zhong 于2020年3月13日周五 下午9:04写道: > > > Hi all, > > > > I would like to start the vote for FLIP-106[1] which is discussed and > > reached consensus in the

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

2020-03-16 Thread jincheng sun
+1 Best, Jincheng Wei Zhong 于2020年3月13日周五 下午9:04写道: > Hi all, > > I would like to start the vote for FLIP-106[1] which is discussed and > reached consensus in the discussion thread[2]. > > The vote will be open for at least 72 hours. I'll try to close it by > 2020-03-18 14:00 UTC, unless

Re: [VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

2020-03-16 Thread jincheng sun
+1 Best, Jincheng Hequn Cheng 于2020年3月16日周一 上午10:01写道: > Hi everyone, > > I'd like to start the vote of FLIP-112[1] which is discussed and reached > consensus in the discussion thread[2]. > The vote will be open for at least 72 hours. Unless there is an objection, > I will try to close it by

Re: Flink YARN app terminated before the client receives the result

2020-03-16 Thread tison
edit: previously after the cancellation we have a longer call chain to #jobReachedGloballyTerminalState which does the archive job & JM graceful showdown, which might take some time so that ... Best, tison. tison 于2020年3月17日周二 上午10:13写道: > Hi Weike & Till, > > I agree with Till and it is also

Re: Flink YARN app terminated before the client receives the result

2020-03-16 Thread tison
Hi Weike & Till, I agree with Till and it is also the analysis from my side. However, it seems even if we don't have FLINK-15116, it is still possible that we complete the cancel future but the cluster got shutdown before it properly delivered the response. There is one thing strange that this

[jira] [Created] (FLINK-16625) Extract BootstrapTools#getEnvironmentVariables to a general utility in ConfigurationUtil

2020-03-16 Thread Canbin Zheng (Jira)
Canbin Zheng created FLINK-16625: Summary: Extract BootstrapTools#getEnvironmentVariables to a general utility in ConfigurationUtil Key: FLINK-16625 URL: https://issues.apache.org/jira/browse/FLINK-16625

[jira] [Created] (FLINK-16624) Support to customize annotations for the rest Service on Kubernetes

2020-03-16 Thread Canbin Zheng (Jira)
Canbin Zheng created FLINK-16624: Summary: Support to customize annotations for the rest Service on Kubernetes Key: FLINK-16624 URL: https://issues.apache.org/jira/browse/FLINK-16624 Project: Flink

[jira] [Created] (FLINK-16623) add the shorthand 'desc' for describe on sql client

2020-03-16 Thread Jun Zhang (Jira)
Jun Zhang created FLINK-16623: - Summary: add the shorthand 'desc' for describe on sql client Key: FLINK-16623 URL: https://issues.apache.org/jira/browse/FLINK-16623 Project: Flink Issue Type:

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

2020-03-16 Thread Till Rohrmann
If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html Cheers,

Re: Flink YARN app terminated before the client receives the result

2020-03-16 Thread Till Rohrmann
Hi Weike, could you share the complete logs with us? Attachments are being filtered out by the Apache mail server but it works if you upload the logs somewhere (e.g. https://gist.github.com/) and then share the link with us. Ideally you run the cluster with DEBUG log settings. I assume that you

[DISCUSS] FLIP-95: New TableSource and TableSink interfaces

2020-03-16 Thread Timo Walther
Hi everyone, I'm happy to present the results of long discussions that we had internally. Jark, Dawid, Aljoscha, Kurt, Jingsong, me, and many more have contributed to this design document. We would like to propose new long-term table source and table sink interfaces:

Re: FLIP-117: HBase catalog

2020-03-16 Thread Bowen Li
Hi, I think core of the jira right now is to investigate if catalogs of schemaless systems like HBase and Elasticsearch bring practical value to users. I haven't used these SQL connectors before, and thus don't have much to say in this case. Can anyone describe how it would work? Maybe @Yu or

[jira] [Created] (FLINK-16622) Type ARRAY of table field 'id' does not match with the physical type LEGACY('ARRAY', 'ANY<[Ljava.lang.String;, rO0ABXNyAD......>') of the 'id' field of the Table

2020-03-16 Thread gkgkgk (Jira)
gkgkgk created FLINK-16622: -- Summary: Type ARRAY of table field 'id' does not match with the physical type LEGACY('ARRAY', 'ANY<[Ljava.lang.String;, rO0ABXNyAD..>') of the 'id' field of the TableSource return type. Key: FLINK-16622

[jira] [Created] (FLINK-16621) Add support for custom logging configuration in docker-flink

2020-03-16 Thread Michal Lyson (Jira)
Michal Lyson created FLINK-16621: Summary: Add support for custom logging configuration in docker-flink Key: FLINK-16621 URL: https://issues.apache.org/jira/browse/FLINK-16621 Project: Flink

[jira] [Created] (FLINK-16620) Add attempt information in logging

2020-03-16 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-16620: -- Summary: Add attempt information in logging Key: FLINK-16620 URL: https://issues.apache.org/jira/browse/FLINK-16620 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-16619) Misleading SlotManagerImpl logging for slot reports of unknown task manager

2020-03-16 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-16619: Summary: Misleading SlotManagerImpl logging for slot reports of unknown task manager Key: FLINK-16619 URL: https://issues.apache.org/jira/browse/FLINK-16619

[jira] [Created] (FLINK-16618) Cannot use Hive GenericUDFBaseNumeric with Flink

2020-03-16 Thread Rui Li (Jira)
Rui Li created FLINK-16618: -- Summary: Cannot use Hive GenericUDFBaseNumeric with Flink Key: FLINK-16618 URL: https://issues.apache.org/jira/browse/FLINK-16618 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-16617) Automatic Any unpacking is broken with type deduction.

2020-03-16 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-16617: Summary: Automatic Any unpacking is broken with type deduction. Key: FLINK-16617 URL: https://issues.apache.org/jira/browse/FLINK-16617 Project: Flink Issue

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-16 Thread Jingsong Li
Hi Piotr, I am very entangled. Let me re-list the table streaming sink requirements: - In table, maybe 90% sinks are for Hive. The parquet and orc are the most important formats. Hive provide RecordWriters, it is easy to support all hive formats by using it, and we don't need concern hive

Re: [DISCUSS] Drop Bucketing Sink

2020-03-16 Thread Robert Metzger
Thank you all for your feedback. I will try to fix the test then (or disable it). Here's a ticket for dropping the BucketingSink: https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever we consider necessary as a "depends on" ticket. @David / @Seth: Where are the tickets depending

[jira] [Created] (FLINK-16616) Drop BucketingSink

2020-03-16 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-16616: -- Summary: Drop BucketingSink Key: FLINK-16616 URL: https://issues.apache.org/jira/browse/FLINK-16616 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-16615) Introduce data structures and utilities to calculate Job Manager memory components

2020-03-16 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-16615: --- Summary: Introduce data structures and utilities to calculate Job Manager memory components Key: FLINK-16615 URL: https://issues.apache.org/jira/browse/FLINK-16615

[jira] [Created] (FLINK-16614) FLIP-116 Unified Memory Configuration for Job Manager

2020-03-16 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-16614: --- Summary: FLIP-116 Unified Memory Configuration for Job Manager Key: FLINK-16614 URL: https://issues.apache.org/jira/browse/FLINK-16614 Project: Flink

[jira] [Created] (FLINK-16613) Builds on Azure pipeline are not triggered

2020-03-16 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-16613: -- Summary: Builds on Azure pipeline are not triggered Key: FLINK-16613 URL: https://issues.apache.org/jira/browse/FLINK-16613 Project: Flink Issue Type:

Re: SerializableHadoopConfiguration

2020-03-16 Thread Till Rohrmann
Hi Sivaprasanna, do you want to collect the set of Hadoop utility classes which could be moved to a flink-hadoop-utils module and start a discuss thread about it? I think this could be a first good step into cleaning up the module structure a bit. Cheers, Till On Fri, Mar 6, 2020 at 7:27 AM

[jira] [Created] (FLINK-16612) Submit job through the rest api, job name will be lost

2020-03-16 Thread Junli Zhang (Jira)
Junli Zhang created FLINK-16612: --- Summary: Submit job through the rest api, job name will be lost Key: FLINK-16612 URL: https://issues.apache.org/jira/browse/FLINK-16612 Project: Flink Issue

[jira] [Created] (FLINK-16611) Datadog reporter should chunk large reports

2020-03-16 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-16611: Summary: Datadog reporter should chunk large reports Key: FLINK-16611 URL: https://issues.apache.org/jira/browse/FLINK-16611 Project: Flink Issue

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-16 Thread Arvid Heise
Voting period is now over even with the roadmap changes (forgot to close on Friday because of all the Coronavirus chaos). We have 4 binding votes (Thomas, Yu, Piotr, Zhijiang) and no objections, so FLIP-76 passed. Thank you very much for your feedback. On Fri, Mar 13, 2020 at 11:08 AM Yu Li

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-16 Thread Andrey Zagrebin
Thanks for the further feedback Thomas and Yangze. > A generic, dynamic configuration mechanism based on environment variables is essential and it is already supported via envsubst and an environment variable that can supply a configuration fragment True, we already have this. As I understand

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-16 Thread Piotr Nowojski
Hi Jingsong, > First way is reusing Batch sink in FLINK-14254, It has handled the partition > and metastore logic well. > - unify batch and streaming > - Using FileOutputFormat is consistent with FileInputFormat. > - Add exactly-once related logic. Just 200+ lines code. > - It's natural to

[jira] [Created] (FLINK-16610) FlinkKafkaInternalProducerITCase fails with timeout exception

2020-03-16 Thread Zhijiang (Jira)
Zhijiang created FLINK-16610: Summary: FlinkKafkaInternalProducerITCase fails with timeout exception Key: FLINK-16610 URL: https://issues.apache.org/jira/browse/FLINK-16610 Project: Flink Issue