[jira] [Created] (FLINK-16300) Reworks SchedulerTestUtils with testing classes to replace mockito usages

2020-02-26 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-16300: --- Summary: Reworks SchedulerTestUtils with testing classes to replace mockito usages Key: FLINK-16300 URL: https://issues.apache.org/jira/browse/FLINK-16300 Project: Flink

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-26 Thread Yadong Xie
Hi Till We keep response a flattened SubtaskTimeInfo and an array of SubtaskTimeInfo to keep the restAPI backward compatible, since Flink users may still need the API('/jobs/{jobId}/vertices/{vertexId}/subtasktimes') to get the flattened SubtaskTimeInfo. If we want to get an array of

[jira] [Created] (FLINK-16299) Release containers recovered from previous attempt in which TaskExecutor is not started.

2020-02-26 Thread Xintong Song (Jira)
Xintong Song created FLINK-16299: Summary: Release containers recovered from previous attempt in which TaskExecutor is not started. Key: FLINK-16299 URL: https://issues.apache.org/jira/browse/FLINK-16299

[VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

2020-02-26 Thread godfrey he
Hi everyone, I'd like to start the vote of FLIP-84[1], which proposes to deprecate some old APIs and introduce some new APIs in TableEnvironment. This FLIP is discussed and reached consensus in the discussion thread[2]. The vote will be open for at least 72 hours. Unless there is an objection, I

[jira] [Created] (FLINK-16298) GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis

2020-02-26 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-16298: --- Summary: GroupWindowTableAggregateITCase.testEventTimeTumblingWindow fails on Travis Key: FLINK-16298 URL: https://issues.apache.org/jira/browse/FLINK-16298 Project:

[jira] [Created] (FLINK-16297) Remove the redundant intent and blank line in highlight code blocks

2020-02-26 Thread Yangze Guo (Jira)
Yangze Guo created FLINK-16297: -- Summary: Remove the redundant intent and blank line in highlight code blocks Key: FLINK-16297 URL: https://issues.apache.org/jira/browse/FLINK-16297 Project: Flink

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-26 Thread Yuan Mei
Congrats! Best Yuan On Thu, Feb 27, 2020 at 8:48 AM Guowei Ma wrote: > Congratulations !! > Best, > Guowei > > > Yun Tang 于2020年2月27日周四 上午2:11写道: > > > Congratulations and well deserved! > > > > > > Best > > Yun Tang > > > > From: Canbin Zheng > > Sent:

[jira] [Created] (FLINK-16296) Improve performance of BaseRowSerializer#serialize() for GenericRow

2020-02-26 Thread Jark Wu (Jira)
Jark Wu created FLINK-16296: --- Summary: Improve performance of BaseRowSerializer#serialize() for GenericRow Key: FLINK-16296 URL: https://issues.apache.org/jira/browse/FLINK-16296 Project: Flink

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jingsong Li
Hi Jark, The matrix I see is SQL cast. If we need bring another conversion matrix that is different from SQL cast, I don't understand the benefits. It makes me difficult to understand. And It seems bad to change the timestamp of different time zones to the same value silently. I have seen a lot

[jira] [Created] (FLINK-16295) Optimize BinaryString.copy to not materialize if there is javaObject

2020-02-26 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-16295: Summary: Optimize BinaryString.copy to not materialize if there is javaObject Key: FLINK-16295 URL: https://issues.apache.org/jira/browse/FLINK-16295 Project: Flink

[jira] [Created] (FLINK-16294) JDBC connector support create database table automatically

2020-02-26 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-16294: -- Summary: JDBC connector support create database table automatically Key: FLINK-16294 URL: https://issues.apache.org/jira/browse/FLINK-16294 Project: Flink Issue

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-26 Thread Guowei Ma
Congratulations !! Best, Guowei Yun Tang 于2020年2月27日周四 上午2:11写道: > Congratulations and well deserved! > > > Best > Yun Tang > > From: Canbin Zheng > Sent: Monday, February 24, 2020 16:07 > To: dev > Subject: Re: [ANNOUNCE] Jingsong Lee becomes a Flink

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-26 Thread Yun Tang
Congratulations and well deserved! Best Yun Tang From: Canbin Zheng Sent: Monday, February 24, 2020 16:07 To: dev Subject: Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer Congratulations !! Dawid Wysakowicz 于2020年2月24日周一 下午3:55写道: > Congratulations

[jira] [Created] (FLINK-16293) Document using plugins in Kubernetes

2020-02-26 Thread Niels Basjes (Jira)
Niels Basjes created FLINK-16293: Summary: Document using plugins in Kubernetes Key: FLINK-16293 URL: https://issues.apache.org/jira/browse/FLINK-16293 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-16292) Execute all end to end tests on AZP

2020-02-26 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-16292: -- Summary: Execute all end to end tests on AZP Key: FLINK-16292 URL: https://issues.apache.org/jira/browse/FLINK-16292 Project: Flink Issue Type: Sub-task

Re: Batch Flink Job S3 write performance vs Spark

2020-02-26 Thread Arvid Heise
Exactly. We use the hadoop-fs as an indirection on top of that, but Spark probably does the same. On Wed, Feb 26, 2020 at 3:52 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Thank you (the two systems running on Java and using the same set of > libraries), so from my

Re: [DISCUSS] FLIP-76: Unaligned checkpoints

2020-02-26 Thread Zhijiang
Thanks for the further explanations, Yu! 1. The inflight buffer spilling process is indeed handled asynchronously. While the buffer is not finished spilling, it would not be recycled to reuse again. Your understanding is right. I guess I misunderstood your previous concern of additional memory

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-26 Thread Till Rohrmann
Fair enough. If this should become a problem we could introduce it later as well. What about changing the SubtasksTimeInfo response type into an array of SubtaskTimeInfo? At the moment SubtasksTimeInfo contains a flattened SubtaskTimeInfo and and array of SubtaskTimeInfo for the previous

Re: Batch Flink Job S3 write performance vs Spark

2020-02-26 Thread Arvid Heise
Fair benchmarks are notoriously difficult to setup. Usually, it's easy to find a workload where one system shines and as its vendor you report that. Then, the competitor benchmarks a different use case where his system outperforms ours. In the end, customers are more confused than before. You

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jark Wu
Hi Jingsong, I don't think it should follow SQL CAST semantics, because it is out of SQL, it happens in connectors which converts users'/external's format into SQL types. I also doubt "timestampFormat" may not work in some cases, because the timestamp format maybe various and mixed in a topic.

[jira] [Created] (FLINK-16291) Count(*) doesn't work with Hive module

2020-02-26 Thread Rui Li (Jira)
Rui Li created FLINK-16291: -- Summary: Count(*) doesn't work with Hive module Key: FLINK-16291 URL: https://issues.apache.org/jira/browse/FLINK-16291 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-16290) HttpUrl might return NULL if a schema is missing

2020-02-26 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-16290: Summary: HttpUrl might return NULL if a schema is missing Key: FLINK-16290 URL: https://issues.apache.org/jira/browse/FLINK-16290 Project: Flink Issue Type:

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jingsong Li
Thanks all for your discussion. Hi Dawid, +1 to apply the logic of parsing a SQL timestamp literal. I don't fully understand the matrix your list. Should this be the semantics of SQL cast? Do you mean this is implicit cast in JSON parser? I doubt that because these implicit casts are not

[jira] [Created] (FLINK-16289) Missing serialVersionUID blocks running in Kubernetes.

2020-02-26 Thread Niels Basjes (Jira)
Niels Basjes created FLINK-16289: Summary: Missing serialVersionUID blocks running in Kubernetes. Key: FLINK-16289 URL: https://issues.apache.org/jira/browse/FLINK-16289 Project: Flink Issue

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-26 Thread Yadong Xie
Hi Till Thanks for your comments. > I have a comment concerning the SubtasksTimesHandler It would be much easier for the frontend to handle a large amount of data if we have a rest API parameter filter, but in my opinion, the attempt list data is not large enough that we have to rely on the

Re: [VOTE] FLIP-104: Add More Metrics to Jobmanager

2020-02-26 Thread lining jing
Hi till, thanks for your reply. > Concerning FLINK-9741, I'm not sure whether we need to fix this issue > before starting this effort. The JobManager's are now running as part of > the cluster entrypoint process for which we should actually report the > metrics (memory usage). I have confirmed

[jira] [Created] (FLINK-16288) Setting the TTL for discarding task pods on Kubernetes.

2020-02-26 Thread Niels Basjes (Jira)
Niels Basjes created FLINK-16288: Summary: Setting the TTL for discarding task pods on Kubernetes. Key: FLINK-16288 URL: https://issues.apache.org/jira/browse/FLINK-16288 Project: Flink

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jark Wu
Hi Dawid, I agree with you. If we want to loosen the format constraint, the important piece is the conversion matrix. The conversion matrix you listed makes sense to me. From my understanding, there should be 6 combination. We can add WITHOUT TIMEZONE => WITHOUT TIMEZONE and WITH TIMEZONE =>

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

2020-02-26 Thread Yadong Xie
Hi Till Thanks a lot for your response > 2. I'm not entirely sure whether I would split the memory ... Split the memory display comes from the 'ancient' design of the web, it is ok for me to change it following total/heap/managed/network/direct/jvm overhead/mapped sequence > 3. Displaying the

[jira] [Created] (FLINK-16287) ES6 sql jar relocates log4j2

2020-02-26 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-16287: Summary: ES6 sql jar relocates log4j2 Key: FLINK-16287 URL: https://issues.apache.org/jira/browse/FLINK-16287 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

2020-02-26 Thread lining jing
Hi Till Thanks for your response! I'm responsible for the RestAPI design part of FLIP-102 1. Some of the newly introduced metrics involve computations on the > TaskManager. I would like to avoid additional computations introduced by > metrics as much as possible because metrics should not affect

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Dawid Wysakowicz
Hi all, @NiYanchun Thank you for reporting this. Yes I think we could improve the behaviour of the JSON format. @Jark First of all I do agree we could/should improve the "user-friendliness" of the JSON format (and unify the behavior across text based formats). I am not sure though if it is as