Re: [DISCUSS] Best practice to run flink on kubernetes

2019-08-09 Thread Jeff Zhang
Thanks Yang. K8s natively integration is very necessary and important for the adoption of flink IMO. I notice that the design doc is written in 2018, is there any changes or update ? >>> Download the flink release binary and create the ~/.kube/config file corresponding to the k8s cluster. It is al

Watermarking in Src and Timestamping downstream

2019-08-09 Thread Roshan Naik
Have streaming use cases where it is useful & easier to generate the watermark in the Source (via ctx.emitWatermark() ) and assign timestamps in a downstream custom operator which calls  output.collect(new StreamRecord(msg, time)). When doing so, I see that the watermark reaches the downstream op

[jira] [Created] (FLINK-13677) Translate "Monitoring Back Pressure" page into Chinese

2019-08-09 Thread fanrui (JIRA)
fanrui created FLINK-13677: -- Summary: Translate "Monitoring Back Pressure" page into Chinese Key: FLINK-13677 URL: https://issues.apache.org/jira/browse/FLINK-13677 Project: Flink Issue Type: Sub-ta

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Xuefu Z
Thanks all for sharing thoughts. I agree with Tzu-li that release schedule deserves a separate discussion. For the PR in question, I'd concur with Till that we keep it due to its limited scope of impact. However, it would be okay to me revert it if it's found to pose a big risk to the stability of

Re: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release

2019-08-09 Thread Tzu-Li (Gordon) Tai
Hi, Update: voting on RC2 for Apache Flink 1.9.0 has started: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Release-1-9-0-release-candidate-2-td31542.html Cheers, Gordon On Thu, Aug 8, 2019 at 5:40 PM Fabian Hueske wrote: > OK, let's stop the discussion about

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-09 Thread Tzu-Li (Gordon) Tai
Please note that the unresolved issues that are still tagged with a fix version "1.9.0", as seen in the JIRA release notes [1], are issues to update documents for new features. I've left them still associated with 1.9.0 since these should still be updated for 1.9.0 soon along with the official rele

[VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-09 Thread Tzu-Li (Gordon) Tai
Hi all, Release candidate #2 for Apache Flink 1.9.0 is now ready for your review. This is the first voting candidate for 1.9.0, following the preview candidates RC0 and RC1. Please review and vote on release candidate #2 for version 1.9.0, as follows: [ ] +1, Approve the release [ ] -1, Do not ap

[jira] [Created] (FLINK-13676) Add class of Vector to Columns mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13676: --- Summary: Add class of Vector to Columns mapper Key: FLINK-13676 URL: https://issues.apache.org/jira/browse/FLINK-13676 Project: Flink Issue Type: Sub-task Co

[jira] [Created] (FLINK-13675) Add class of Vector Slice Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13675: --- Summary: Add class of Vector Slice Mapper Key: FLINK-13675 URL: https://issues.apache.org/jira/browse/FLINK-13675 Project: Flink Issue Type: Sub-task Compone

[jira] [Created] (FLINK-13674) Add class of Vector Size Hint Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13674: --- Summary: Add class of Vector Size Hint Mapper Key: FLINK-13674 URL: https://issues.apache.org/jira/browse/FLINK-13674 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-13673) Add class of Vector Normalize Mapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13673: --- Summary: Add class of Vector Normalize Mapper Key: FLINK-13673 URL: https://issues.apache.org/jira/browse/FLINK-13673 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-13672) Add class for VectorInteractionMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13672: --- Summary: Add class for VectorInteractionMapper Key: FLINK-13672 URL: https://issues.apache.org/jira/browse/FLINK-13672 Project: Flink Issue Type: Sub-task Co

[jira] [Created] (FLINK-13671) Add class for VectorEleWiseProductMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13671: --- Summary: Add class for VectorEleWiseProductMapper Key: FLINK-13671 URL: https://issues.apache.org/jira/browse/FLINK-13671 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13670) Add class for VectorAssemblerMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13670: --- Summary: Add class for VectorAssemblerMapper Key: FLINK-13670 URL: https://issues.apache.org/jira/browse/FLINK-13670 Project: Flink Issue Type: Sub-task Comp

[jira] [Created] (FLINK-13669) Add class for BinarizerMapper

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13669: --- Summary: Add class for BinarizerMapper Key: FLINK-13669 URL: https://issues.apache.org/jira/browse/FLINK-13669 Project: Flink Issue Type: Sub-task Components

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Till Rohrmann
I can only second Stephan's, Timo's, Piotr's and Gordon's points. I also want to re-emphasize the importance of sticking to the agreed processes for the community. If people merge features at free will after the feature freeze, then they willingly put the release and with that the work of the whol

[DISCUSS] Best practice to run flink on kubernetes

2019-08-09 Thread Yang Wang
Hi all, Currently cloud native architectures has been introduced to many companies in production. They use kubernetes to run deep learning, web server, etc. If we could deploy the per-job/session flink cluster on kubernetes to make it mix-run with other workloads, the cluster resource utilization

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-09 Thread Tzu-Li (Gordon) Tai
Hi, Just as a reminder, I think we should not focus here on discussing whether or not time-based releases is a suitable release model for us. That doesn’t seem to be the original intent of this thread, and is therefore more suited for a separate discussion with the community, especially taking int

[jira] [Created] (FLINK-13668) Add abstract classes for three typical scenarios of (Flat)Mapper.

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13668: --- Summary: Add abstract classes for three typical scenarios of (Flat)Mapper. Key: FLINK-13668 URL: https://issues.apache.org/jira/browse/FLINK-13668 Project: Flink Issu

[jira] [Created] (FLINK-13667) Add the utility class for the Table

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13667: --- Summary: Add the utility class for the Table Key: FLINK-13667 URL: https://issues.apache.org/jira/browse/FLINK-13667 Project: Flink Issue Type: Sub-task Comp

[jira] [Created] (FLINK-13666) Add an implement of collector with the row type

2019-08-09 Thread Xu Yang (JIRA)
Xu Yang created FLINK-13666: --- Summary: Add an implement of collector with the row type Key: FLINK-13666 URL: https://issues.apache.org/jira/browse/FLINK-13666 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13665) decimal(p, s) where p is less than s should be illegal

2019-08-09 Thread Zhenghua Gao (JIRA)
Zhenghua Gao created FLINK-13665: Summary: decimal(p, s) where p is less than s should be illegal Key: FLINK-13665 URL: https://issues.apache.org/jira/browse/FLINK-13665 Project: Flink Issue

[jira] [Created] (FLINK-13664) "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING

2019-08-09 Thread Zhenghua Gao (JIRA)
Zhenghua Gao created FLINK-13664: Summary: "MD5" and "SHA" functions should SqlTypeFamily.CHARACTER instend of SqlTypeFamily.STRING Key: FLINK-13664 URL: https://issues.apache.org/jira/browse/FLINK-13664

Re: [DISCUSS] Repository split

2019-08-09 Thread Biao Liu
Hi folks, Thanks for bringing this discussion Chesnay. +1 for the motivation. It's really a bad experience of waiting Travis building for a long time. WRT the solution, personally I agree with Dawid/David. IMO the biggest benefit of splitting repository is reducing build time. I think it could

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

2019-08-09 Thread Xintong Song
Thanks for the feedback, Yang. Regarding your comments: *Native and Direct Memory* I think setting a very large max direct memory size definitely has some good sides. E.g., we do not worry about direct OOM, and we don't even need to allocate managed / network memory with Unsafe.allocate() . Howev

[jira] [Created] (FLINK-13663) SQL Client end-to-end test for modern Kafka failed on Travis

2019-08-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-13663: - Summary: SQL Client end-to-end test for modern Kafka failed on Travis Key: FLINK-13663 URL: https://issues.apache.org/jira/browse/FLINK-13663 Project: Flink

[jira] [Created] (FLINK-13662) FlinkKinesisProducerTest.testBackpressure failed on Travis

2019-08-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-13662: - Summary: FlinkKinesisProducerTest.testBackpressure failed on Travis Key: FLINK-13662 URL: https://issues.apache.org/jira/browse/FLINK-13662 Project: Flink