[jira] [Created] (FLINK-11458) Ensure sink commit side-effects when cancelling with savepoint.

2019-01-29 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-11458: -- Summary: Ensure sink commit side-effects when cancelling with savepoint. Key: FLINK-11458 URL: https://issues.apache.org/jira/browse/FLINK-11458 Project: Flink

[jira] [Created] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio

2019-01-29 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-11459: --- Summary: Presto S3 does not show errors due to missing credentials with minio Key: FLINK-11459 URL: https://issues.apache.org/jira/browse/FLINK-11459 Project: Flink

Re: Side Outputs for late arriving records

2019-01-29 Thread Fabian Hueske
Hi Ramya, If you don't want to apply any logic but just filter late records, you should not use a window because it needs to shuffle and group records into windows. Instead, you can use a non-keyed ProcessFunction and compare the timestamp of the record (context.timestamp()) with the current

Re: [DISCUSS] Towards a leaner flink-dist

2019-01-29 Thread Chesnay Schepler
It is not viable for us, as of right now, to release both a lean and fat version of flink-dist. We don't have the required tooling to assemble a correct NOTICE file for that scenario. Besides that his would also go against recent efforts to reduce the total size of a Flink release, as we'd be

Re: [DISCUSS] Improvement to Flink Window Operator with Slicing

2019-01-29 Thread Fabian Hueske
Thank you Rong! The performance of sliding windows is an issue for many users. Adding support for a more efficient window is a great effort. Thank you, Fabian Am Di., 29. Jan. 2019 um 16:37 Uhr schrieb Rong Rong : > Hi all, > > Thanks for the feedbacks and suggestions to the design doc. I have

[jira] [Created] (FLINK-11453) Support SliceWindow with forwardable pane info

2019-01-29 Thread Rong Rong (JIRA)
Rong Rong created FLINK-11453: - Summary: Support SliceWindow with forwardable pane info Key: FLINK-11453 URL: https://issues.apache.org/jira/browse/FLINK-11453 Project: Flink Issue Type:

[jira] [Created] (FLINK-11454) Support MergedStream operation

2019-01-29 Thread Rong Rong (JIRA)
Rong Rong created FLINK-11454: - Summary: Support MergedStream operation Key: FLINK-11454 URL: https://issues.apache.org/jira/browse/FLINK-11454 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-11455) Support evictor operations on slicing and merging operators

2019-01-29 Thread Rong Rong (JIRA)
Rong Rong created FLINK-11455: - Summary: Support evictor operations on slicing and merging operators Key: FLINK-11455 URL: https://issues.apache.org/jira/browse/FLINK-11455 Project: Flink Issue

[jira] [Created] (FLINK-11457) PrometheusPushGatewayReporter either overwrites its own metrics or creates too may labels

2019-01-29 Thread Oscar Westra van Holthe - Kind (JIRA)
Oscar Westra van Holthe - Kind created FLINK-11457: -- Summary: PrometheusPushGatewayReporter either overwrites its own metrics or creates too may labels Key: FLINK-11457 URL:

[jira] [Created] (FLINK-11456) Improve window operator with sliding window assigners

2019-01-29 Thread Rong Rong (JIRA)
Rong Rong created FLINK-11456: - Summary: Improve window operator with sliding window assigners Key: FLINK-11456 URL: https://issues.apache.org/jira/browse/FLINK-11456 Project: Flink Issue Type:

Re: [DISCUSS] Improvement to Flink Window Operator with Slicing

2019-01-29 Thread Rong Rong
Hi all, Thanks for the feedbacks and suggestions to the design doc. I have created a parent JIRA [1] to track the related tasks and started the implementation process Any further feedbacks or suggestions are highly appreciated. Best, Rong [1] https://issues.apache.org/jira/browse/FLINK-11276

Re: [DISCUSS] Proposal of external shuffle service

2019-01-29 Thread qi luo
Very clear. Thanks! > On Jan 28, 2019, at 10:29 PM, zhijiang wrote: > > Hi Qi, > > Thanks for the concerns of this proposal. In Blink we implemented the > YarnShuffleService which is mainly used for batch jobs in production and some > benchmark before. This YarnShuffleService is not within

Re: Ratelimiting in the Flink Kafka connector

2019-01-29 Thread Thomas Weise
It is preferred for the service to rate limit. The problem is that not all Kafka setups have that control enabled / support for it. Even when rate limiting was enabled, it may still be *nice* for the client to gracefully handle it. There was discussion in the past that we should not bloat the

Re: Flink Table Case Statements appending spaces

2019-01-29 Thread Hequn Cheng
Hi Ramya, Fabian is right. The behavior in strict SQL standard mode(SQL:2003) returns a CHAR(N) type with blank-padded. Best, Hequn On Wed, Jan 30, 2019 at 1:59 AM Fabian Hueske wrote: > Hi, > > Table API queries are executed with SQL semantics. > In SQL, the strings are padded because the

[jira] [Created] (FLINK-11460) Remove useless AcknowledgeStreamMockEnvironment

2019-01-29 Thread zhijiang (JIRA)
zhijiang created FLINK-11460: Summary: Remove useless AcknowledgeStreamMockEnvironment Key: FLINK-11460 URL: https://issues.apache.org/jira/browse/FLINK-11460 Project: Flink Issue Type: Task

?????? [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread ????
Hi Jark, Thanks for starting the discussion! I think integrating the website and documentation as subdirectories into the existing website and docs is a very good approach too. And when submitting a pull request, we should notice the author whether they changed the documents, If yes, should

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread Jörn Franke
Keep also in mind the other direction eg a new/modified version of the Chinese documentation needs to be reflected in the English one. > Am 29.01.2019 um 06:35 schrieb SteNicholas : > > Hi Jark, > > Thank you for starting this discussion.I am very willing to participate in > flink document

Re: [DISCUSS] Start a user...@flink.apache.org mailing list for the Chinese-speaking community?

2019-01-29 Thread Robert Metzger
Success! The mailing list has been created. Send an email to "user-zh-subscr...@flink.apache.org" to subscribe! I've also updated the website with the list: https://flink.apache.org/community.html I will now also tweet about it, even though I believe it'll be more important to advertise the list

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread JMX
Hi Jark, I am glad to see your exciting email about contributing to Flink Chinese Community. Remebering the days to solve outstanding issues in Spark Officical Documents, the project was developped slowly, I am looking forward to more and more contributions to Flink Chinese Community. Of

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread SteNicholas
Hi Jark, Thank you for starting this discussion.I am very willing to participate in flink document translation. Best, Nicholas -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-11444) Deprecate methods for uncoupling table API and table core

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11444: Summary: Deprecate methods for uncoupling table API and table core Key: FLINK-11444 URL: https://issues.apache.org/jira/browse/FLINK-11444 Project: Flink

[jira] [Created] (FLINK-11446) FlinkKafkaProducer011ITCase.testRecoverCommittedTransaction failed on Travis

2019-01-29 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-11446: - Summary: FlinkKafkaProducer011ITCase.testRecoverCommittedTransaction failed on Travis Key: FLINK-11446 URL: https://issues.apache.org/jira/browse/FLINK-11446

[jira] [Created] (FLINK-11448) Clean-up and prepare Table API to be uncoupled from table core

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11448: Summary: Clean-up and prepare Table API to be uncoupled from table core Key: FLINK-11448 URL: https://issues.apache.org/jira/browse/FLINK-11448 Project: Flink

[jira] [Created] (FLINK-11442) Upgrade OSS SDK Version

2019-01-29 Thread wujinhu (JIRA)
wujinhu created FLINK-11442: --- Summary: Upgrade OSS SDK Version Key: FLINK-11442 URL: https://issues.apache.org/jira/browse/FLINK-11442 Project: Flink Issue Type: Improvement Components:

numLateRecordsDropped

2019-01-29 Thread Ramya Ramamurthy
Hi, I am reading from Kafka and pushing it to the ES. There are basically 2 operators that we have, one to consume from Kafka and the other does the operation on the Flink table and pushes to ES. [image: image.png] I am able to see the Flink Metrics : numLateRecordsDropped on my second

[jira] [Created] (FLINK-11447) Deprecate "new Table(TableEnvironment, String)"

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11447: Summary: Deprecate "new Table(TableEnvironment, String)" Key: FLINK-11447 URL: https://issues.apache.org/jira/browse/FLINK-11447 Project: Flink Issue Type:

[jira] [Created] (FLINK-11449) Uncouple the Expression class from RexNodes

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11449: Summary: Uncouple the Expression class from RexNodes Key: FLINK-11449 URL: https://issues.apache.org/jira/browse/FLINK-11449 Project: Flink Issue Type: New

[jira] [Created] (FLINK-11450) Port and move TableSource and TableSink to flink-table-common

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11450: Summary: Port and move TableSource and TableSink to flink-table-common Key: FLINK-11450 URL: https://issues.apache.org/jira/browse/FLINK-11450 Project: Flink

Re: [DISCUSS] Start a user...@flink.apache.org mailing list for the Chinese-speaking community?

2019-01-29 Thread Jark Wu
Cheers! Subscribed. Looking forward to the first Chinese question ;) On Tue, 29 Jan 2019 at 17:16, Robert Metzger wrote: > Success! > The mailing list has been created. > > Send an email to "user-zh-subscr...@flink.apache.org" to subscribe! > I've also updated the website with the list: >

Re: [DISCUSS] Start new Review Process

2019-01-29 Thread Robert Metzger
@Fabian: Thank you for your suggestions. Multiple approvals in one comment are already possible. I will see if I can easily add multiple approvals in one line as well. I will also address 2) and 3). Regarding usage of the bot: Anyone can use it! It is up to the committer who's merging in the end

[jira] [Created] (FLINK-11443) Recovering from save point error after adding "sum (constant)"

2019-01-29 Thread tivanli (JIRA)
tivanli created FLINK-11443: --- Summary: Recovering from save point error after adding "sum (constant)" Key: FLINK-11443 URL: https://issues.apache.org/jira/browse/FLINK-11443 Project: Flink Issue

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-29 Thread Timo Walther
Hi everyone, thanks for the positive feedback we received so far. I converted the design document into a actual FLIP in the wiki. You can find it here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions I'm in the process of creating

[jira] [Created] (FLINK-11445) Deprecate static methods in TableEnvironments

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11445: Summary: Deprecate static methods in TableEnvironments Key: FLINK-11445 URL: https://issues.apache.org/jira/browse/FLINK-11445 Project: Flink Issue Type:

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread Chesnay Schepler
Is the build-system for flink-china.org identical to flink-web? On 28.01.2019 12:48, Jark Wu wrote: Hi all, In the past year, the Chinese community is working on building a Chinese translated Flink website (http://flink.apache.org) and documents (

Re: Side Outputs for late arriving records

2019-01-29 Thread Fabian Hueske
Hi Ramya, This works by calling getSideOutput() on the main output of the window function. The main output is collected by applying a function on the window. DataStream input = ... OutputTag lateTag = ... DataStream mainResult = input .keyBy(...) .window(...) .sideOutputLateData(lateTag)

Re: Side Outputs for late arriving records

2019-01-29 Thread Ramya Ramamurthy
Hi Fabian, I do not have anything to do "yourfunction". .apply windowFunction is legacy is what the documentation says. But I am at a loss to understand which of the reduce, aggregate, Fold, Apply must i use, as i hardly have any operations to perform but to return the stream with no late data

Flink Table Case Statements appending spaces

2019-01-29 Thread Ramya Ramamurthy
Hi, I have encountered a weird issue. When constructing a Table Query with CASE Statements, like below: .append("CASE ") .append("WHEN aggrcat = '0' AND botcode='r4' THEN 'monitoring' ") .append("WHEN aggrcat = '1' AND botcode='r4' THEN 'aggregator' ") .append("WHEN aggrcat = '2' AND

Re: [DISCUSS] Start new Review Process

2019-01-29 Thread Aljoscha Krettek
What do you mean by “merging cannot happen through the GitHub user interface”? You can in fact merge PRs by clicking on the merge button, or “rebase and merge”. Aljoscha > On 29. Jan 2019, at 11:58, Robert Metzger wrote: > > @Fabian: Thank you for your suggestions. Multiple approvals in one

Re: [DISCUSS] Start new Review Process

2019-01-29 Thread Chesnay Schepler
I assume he means that bots cannot do it, as they'd require committer permissions. Same with assigning reviewers and such. On 29.01.2019 13:09, Aljoscha Krettek wrote: What do you mean by “merging cannot happen through the GitHub user interface”? You can in fact merge PRs by clicking on the

[jira] [Created] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11451: Summary: Move *QueryConfig and TableDescriptor to flink-table-api-java Key: FLINK-11451 URL: https://issues.apache.org/jira/browse/FLINK-11451 Project: Flink

[jira] [Created] (FLINK-11452) Make the table planner pluggable

2019-01-29 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11452: Summary: Make the table planner pluggable Key: FLINK-11452 URL: https://issues.apache.org/jira/browse/FLINK-11452 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Contributing Chinese website and docs to Apache Flink

2019-01-29 Thread Jark Wu
Hi, Thank you all for welcoming this proposal. I will draft a detailed proposal about how to do contribute it step by step in the next few days, and create umbrella JIRA once the proposal is accepted. @Fabian @JIngcheng Regarding the name, I'm fine with https://flink.apache.org/zh/. @Yun Tang