Re: [DISCUSS] Best practice to run flink on kubernetes

2019-08-09 Thread Jeff Zhang
Thanks Yang. K8s natively integration is very necessary and important for the adoption of flink IMO. I notice that the design doc is written in 2018, is there any changes or update ? >>> Download the flink release binary and create the ~/.kube/config file corresponding to the k8s cluster. It is

Help required on Flink config settings

2019-08-09 Thread Karthick Thanigaimani
Hi Team, We are trying to tune our application / flink settings. Could you please advise if there is any best practice or calculator to find out the optimal settings of the Flink system conf file. e.g. by providing the CPU and Memory it should give the settings required. We are using Flink 1.4.

Re: [DISCUSS] Repository split

2019-08-09 Thread Biao Liu
Hi folks, Thanks for bringing this discussion Chesnay. +1 for the motivation. It's really a bad experience of waiting Travis building for a long time. WRT the solution, personally I agree with Dawid/David. IMO the biggest benefit of splitting repository is reducing build time. I think it could

[jira] [Created] (FLINK-13662) FlinkKinesisProducerTest.testBackpressure failed on Travis

2019-08-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-13662: - Summary: FlinkKinesisProducerTest.testBackpressure failed on Travis Key: FLINK-13662 URL: https://issues.apache.org/jira/browse/FLINK-13662 Project: Flink

[jira] [Created] (FLINK-13665) decimal(p, s) where p is less than s should be illegal

2019-08-09 Thread Zhenghua Gao (JIRA)
Zhenghua Gao created FLINK-13665: Summary: decimal(p, s) where p is less than s should be illegal Key: FLINK-13665 URL: https://issues.apache.org/jira/browse/FLINK-13665 Project: Flink Issue

[jira] [Created] (FLINK-13666) Add an implement of collector with the row type

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13666: --- Summary: Add an implement of collector with the row type Key: FLINK-13666 URL: https://issues.apache.org/jira/browse/FLINK-13666 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Flink Docker Playgrounds

2019-08-09 Thread Konstantin Knauf
Hi everyone, Hi Fabian, I am also in favor of option 1. Besides the playgrounds it is a good opportunity to explore this process for official Docker images as Till suggested. This needs a separate discussion, though. Best, Konstantin On Fri, Aug 9, 2019 at 5:25 AM Yang Wang wrote: > Hey

[jira] [Created] (FLINK-13664) "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING

2019-08-09 Thread Zhenghua Gao (JIRA)
Zhenghua Gao created FLINK-13664: Summary: "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING Key: FLINK-13664 URL: https://issues.apache.org/jira/browse/FLINK-13664

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Piotr Nowojski
Hi Xuefu, As others have mentioned, time based releases were decided because of some good reasons (more predictability for users so that they can take that into account for planning next upgrades). That doesn’t mean we can not allow for some flexibility: we do, we have already postponed this

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-08-09 Thread Xintong Song
Thanks for the feedback, Yang. Regarding your comments: *Native and Direct Memory* I think setting a very large max direct memory size definitely has some good sides. E.g., we do not worry about direct OOM, and we don't even need to allocate managed / network memory with Unsafe.allocate() .

[jira] [Created] (FLINK-13663) SQL Client end-to-end test for modern Kafka failed on Travis

2019-08-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-13663: - Summary: SQL Client end-to-end test for modern Kafka failed on Travis Key: FLINK-13663 URL: https://issues.apache.org/jira/browse/FLINK-13663 Project: Flink

Re: [DISCUSS] Repository split

2019-08-09 Thread Piotr Nowojski
Hi, Re Jark’s: > Ad. 2 I can't see how unstable connectors tests can be fixed more quickly > after moved to a separate repositories. It’s more about probability of intermittent failures across all of the modules adding up, causing whole build to fail almost all the time. With separate

[jira] [Created] (FLINK-13669) Add class for BinarizerMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13669: --- Summary: Add class for BinarizerMapper Key: FLINK-13669 URL: https://issues.apache.org/jira/browse/FLINK-13669 Project: Flink Issue Type: Sub-task

[DISCUSS] Best practice to run flink on kubernetes

2019-08-09 Thread Yang Wang
Hi all, Currently cloud native architectures has been introduced to many companies in production. They use kubernetes to run deep learning, web server, etc. If we could deploy the per-job/session flink cluster on kubernetes to make it mix-run with other workloads, the cluster resource utilization

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Till Rohrmann
I can only second Stephan's, Timo's, Piotr's and Gordon's points. I also want to re-emphasize the importance of sticking to the agreed processes for the community. If people merge features at free will after the feature freeze, then they willingly put the release and with that the work of the

[jira] [Created] (FLINK-13668) Add abstract classes for three typical scenarios of (Flat)Mapper.

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13668: --- Summary: Add abstract classes for three typical scenarios of (Flat)Mapper. Key: FLINK-13668 URL: https://issues.apache.org/jira/browse/FLINK-13668 Project: Flink

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Tzu-Li (Gordon) Tai
Hi, Just as a reminder, I think we should not focus here on discussing whether or not time-based releases is a suitable release model for us. That doesn’t seem to be the original intent of this thread, and is therefore more suited for a separate discussion with the community, especially taking

[jira] [Created] (FLINK-13667) Add the utility class for the Table

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13667: --- Summary: Add the utility class for the Table Key: FLINK-13667 URL: https://issues.apache.org/jira/browse/FLINK-13667 Project: Flink Issue Type: Sub-task

[VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-09 Thread Tzu-Li (Gordon) Tai
Hi all, Release candidate #2 for Apache Flink 1.9.0 is now ready for your review. This is the first voting candidate for 1.9.0, following the preview candidates RC0 and RC1. Please review and vote on release candidate #2 for version 1.9.0, as follows: [ ] +1, Approve the release [ ] -1, Do not

[jira] [Created] (FLINK-13673) Add class of Vector Normalize Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13673: --- Summary: Add class of Vector Normalize Mapper Key: FLINK-13673 URL: https://issues.apache.org/jira/browse/FLINK-13673 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13670) Add class for VectorAssemblerMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13670: --- Summary: Add class for VectorAssemblerMapper Key: FLINK-13670 URL: https://issues.apache.org/jira/browse/FLINK-13670 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13671) Add class for VectorEleWiseProductMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13671: --- Summary: Add class for VectorEleWiseProductMapper Key: FLINK-13671 URL: https://issues.apache.org/jira/browse/FLINK-13671 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13672) Add class for VectorInteractionMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13672: --- Summary: Add class for VectorInteractionMapper Key: FLINK-13672 URL: https://issues.apache.org/jira/browse/FLINK-13672 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13674) Add class of Vector Size Hint Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13674: --- Summary: Add class of Vector Size Hint Mapper Key: FLINK-13674 URL: https://issues.apache.org/jira/browse/FLINK-13674 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13675) Add class of Vector Slice Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13675: --- Summary: Add class of Vector Slice Mapper Key: FLINK-13675 URL: https://issues.apache.org/jira/browse/FLINK-13675 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13676) Add class of Vector to Columns mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13676: --- Summary: Add class of Vector to Columns mapper Key: FLINK-13676 URL: https://issues.apache.org/jira/browse/FLINK-13676 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Xuefu Z
Thanks all for sharing thoughts. I agree with Tzu-li that release schedule deserves a separate discussion. For the PR in question, I'd concur with Till that we keep it due to its limited scope of impact. However, it would be okay to me revert it if it's found to pose a big risk to the stability

[jira] [Created] (FLINK-13677) Translate "Monitoring Back Pressure" page into Chinese

2019-08-09 Thread fanrui (JIRA)
fanrui created FLINK-13677: -- Summary: Translate "Monitoring Back Pressure" page into Chinese Key: FLINK-13677 URL: https://issues.apache.org/jira/browse/FLINK-13677 Project: Flink Issue Type:

Watermarking in Src and Timestamping downstream

2019-08-09 Thread Roshan Naik
Have streaming use cases where it is useful & easier to generate the watermark in the Source (via ctx.emitWatermark() ) and assign timestamps in a downstream custom operator which calls  output.collect(new StreamRecord(msg, time)). When doing so, I see that the watermark reaches the downstream