Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Matthias Pohl]

I agree - thanks for managing the release, Dian & Robert! +1

Matthias

On Fri, Dec 11, 2020 at 7:58 AM Yu Li  wrote:

> Hurray! Thanks everyone who contributed and thanks Robert and Dian for
> being our release managers!
>
> Best Regards,
> Yu
>
>
> On Fri, 11 Dec 2020 at 14:56, Congxian Qiu  wrote:
>
> > Thanks Dian and Robert for the great work as release manager, and
> everyone
> > who makes this release possible.
> >
> > Best,
> > Congxian
> >
> >
> > Xintong Song  于2020年12月11日周五 下午1:49写道：
> >
> > > Congratulations~! Thanks Robert, Dian and everyone.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Dec 10, 2020 at 11:23 PM Robert Metzger 
> > > wrote:
> > >
> > > > Thanks Nick!
> > > >
> > > > This is blocked on the maintainers of the official images to merge
> this
> > > PR:
> > > > https://github.com/docker-library/official-images/pull/9249
> > > > I can not really make an estimate when they'll merge it.
> > > >
> > > > On Thu, Dec 10, 2020 at 4:20 PM nick toker  >
> > > > wrote:
> > > >
> > > > > Hi
> > > > > first good job  and tank you
> > > > >
> > > > > i don't find in docker hub the new version 1.12
> > > > >
> > > > > when it will be there ?
> > > > >
> > > > > nick
> > > > >
> > > > > ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> > > > > rmetz...@apache.org‬‏>:‬
> > > > >
> > > > >> The Apache Flink community is very happy to announce the release
> of
> > > > >> Apache Flink 1.12.0, which is the latest major release.
> > > > >>
> > > > >> Apache Flink® is an open-source stream processing framework for
> > > > >> distributed, high-performing, always-available, and accurate data
> > > > streaming
> > > > >> applications.
> > > > >>
> > > > >> The release is available for download at:
> > > > >> https://flink.apache.org/downloads.html
> > > > >>
> > > > >> Please check out the release blog post for an overview of the
> > > > >> improvements for this bugfix release:
> > > > >> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> > > > >>
> > > > >> The full release notes are available in Jira:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> > > > >>
> > > > >> We would like to thank all contributors of the Apache Flink
> > community
> > > > who
> > > > >> made this release possible!
> > > > >>
> > > > >> Regards,
> > > > >> Dian & Robert
> > > > >>
> > > > >>
> > > >
> > >
> >
>


-- 

Matthias Pohl | Engineer

Follow us @VervericaData Ververica 

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

[jira] [Created] (FLINK-20572) HiveCatalog should be a standalone module

[Rui Li (Jira)]

Rui Li created FLINK-20572:
--

 Summary: HiveCatalog should be a standalone module
 Key: FLINK-20572
 URL: https://issues.apache.org/jira/browse/FLINK-20572
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Reporter: Rui Li
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Yu Li]

Hurray! Thanks everyone who contributed and thanks Robert and Dian for
being our release managers!

Best Regards,
Yu


On Fri, 11 Dec 2020 at 14:56, Congxian Qiu  wrote:

> Thanks Dian and Robert for the great work as release manager, and everyone
> who makes this release possible.
>
> Best,
> Congxian
>
>
> Xintong Song  于2020年12月11日周五 下午1:49写道：
>
> > Congratulations~! Thanks Robert, Dian and everyone.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Dec 10, 2020 at 11:23 PM Robert Metzger 
> > wrote:
> >
> > > Thanks Nick!
> > >
> > > This is blocked on the maintainers of the official images to merge this
> > PR:
> > > https://github.com/docker-library/official-images/pull/9249
> > > I can not really make an estimate when they'll merge it.
> > >
> > > On Thu, Dec 10, 2020 at 4:20 PM nick toker 
> > > wrote:
> > >
> > > > Hi
> > > > first good job  and tank you
> > > >
> > > > i don't find in docker hub the new version 1.12
> > > >
> > > > when it will be there ?
> > > >
> > > > nick
> > > >
> > > > ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> > > > rmetz...@apache.org‬‏>:‬
> > > >
> > > >> The Apache Flink community is very happy to announce the release of
> > > >> Apache Flink 1.12.0, which is the latest major release.
> > > >>
> > > >> Apache Flink® is an open-source stream processing framework for
> > > >> distributed, high-performing, always-available, and accurate data
> > > streaming
> > > >> applications.
> > > >>
> > > >> The release is available for download at:
> > > >> https://flink.apache.org/downloads.html
> > > >>
> > > >> Please check out the release blog post for an overview of the
> > > >> improvements for this bugfix release:
> > > >> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> > > >>
> > > >> The full release notes are available in Jira:
> > > >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> > > >>
> > > >> We would like to thank all contributors of the Apache Flink
> community
> > > who
> > > >> made this release possible!
> > > >>
> > > >> Regards,
> > > >> Dian & Robert
> > > >>
> > > >>
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Congxian Qiu]

Thanks Dian and Robert for the great work as release manager, and everyone
who makes this release possible.

Best,
Congxian


Xintong Song  于2020年12月11日周五 下午1:49写道：

> Congratulations~! Thanks Robert, Dian and everyone.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Dec 10, 2020 at 11:23 PM Robert Metzger 
> wrote:
>
> > Thanks Nick!
> >
> > This is blocked on the maintainers of the official images to merge this
> PR:
> > https://github.com/docker-library/official-images/pull/9249
> > I can not really make an estimate when they'll merge it.
> >
> > On Thu, Dec 10, 2020 at 4:20 PM nick toker 
> > wrote:
> >
> > > Hi
> > > first good job  and tank you
> > >
> > > i don't find in docker hub the new version 1.12
> > >
> > > when it will be there ?
> > >
> > > nick
> > >
> > > ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> > > rmetz...@apache.org‬‏>:‬
> > >
> > >> The Apache Flink community is very happy to announce the release of
> > >> Apache Flink 1.12.0, which is the latest major release.
> > >>
> > >> Apache Flink® is an open-source stream processing framework for
> > >> distributed, high-performing, always-available, and accurate data
> > streaming
> > >> applications.
> > >>
> > >> The release is available for download at:
> > >> https://flink.apache.org/downloads.html
> > >>
> > >> Please check out the release blog post for an overview of the
> > >> improvements for this bugfix release:
> > >> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> > >>
> > >> The full release notes are available in Jira:
> > >>
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> > >>
> > >> We would like to thank all contributors of the Apache Flink community
> > who
> > >> made this release possible!
> > >>
> > >> Regards,
> > >> Dian & Robert
> > >>
> > >>
> >
>

[jira] [Created] (FLINK-20571) Add dynamic open/close LatencyMarksEmitter to support online debug and monitoring

[zlzhang0122 (Jira)]

zlzhang0122 created FLINK-20571:
---

 Summary: Add dynamic open/close LatencyMarksEmitter to support 
online debug and monitoring
 Key: FLINK-20571
 URL: https://issues.apache.org/jira/browse/FLINK-20571
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: zlzhang0122


Now, flink has provided latency metrics to monitor the latency, but this 
function mainly used in debugging contexts rather than in production contexts 
because of throughput effect. If we can provider an api that can dynamic 
open/close this function, then we can monitor the online data latency and find 
out the performance bottleneck in time with out restart the job, which maybe 
very helpful. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20570) `/docs/stream/operators/process_function.md`, line 252. The `NOTE tip` css style is different from the others.

[shizhengchao (Jira)]

shizhengchao created FLINK-20570:


 Summary:  `/docs/stream/operators/process_function.md`, line 252. 
The `NOTE tip` css style is different from the others.
 Key: FLINK-20570
 URL: https://issues.apache.org/jira/browse/FLINK-20570
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.12.0
Reporter: shizhengchao


in `/docs/stream/operators/process_function.md`, line 252.  The `NOTE` css 
style is different from the others.
{code:java}
current is: **NOTE:**

and another style is : Note
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Xintong Song]

Congratulations~! Thanks Robert, Dian and everyone.

Thank you~

Xintong Song



On Thu, Dec 10, 2020 at 11:23 PM Robert Metzger  wrote:

> Thanks Nick!
>
> This is blocked on the maintainers of the official images to merge this PR:
> https://github.com/docker-library/official-images/pull/9249
> I can not really make an estimate when they'll merge it.
>
> On Thu, Dec 10, 2020 at 4:20 PM nick toker 
> wrote:
>
> > Hi
> > first good job  and tank you
> >
> > i don't find in docker hub the new version 1.12
> >
> > when it will be there ?
> >
> > nick
> >
> > ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> > rmetz...@apache.org‬‏>:‬
> >
> >> The Apache Flink community is very happy to announce the release of
> >> Apache Flink 1.12.0, which is the latest major release.
> >>
> >> Apache Flink® is an open-source stream processing framework for
> >> distributed, high-performing, always-available, and accurate data
> streaming
> >> applications.
> >>
> >> The release is available for download at:
> >> https://flink.apache.org/downloads.html
> >>
> >> Please check out the release blog post for an overview of the
> >> improvements for this bugfix release:
> >> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> >>
> >> The full release notes are available in Jira:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> >>
> >> We would like to thank all contributors of the Apache Flink community
> who
> >> made this release possible!
> >>
> >> Regards,
> >> Dian & Robert
> >>
> >>
>

Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

[Zhijiang]

Thanks for the further info and explanations! I have no other concerns.

Best,
Zhijiang


--
From:Rui Li 
Send Time:2020年12月10日(星期四) 20:35
To:dev ; Zhijiang 
Subject:Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

Hi Zhijiang,

Glad to know you're interested in this FLIP. I wouldn't claim 100%
compatibility with this FLIP. That's because Flink doesn't have the
functionalities to support all Hive's features. To list a few examples:

   1. Hive allows users to process data with shell scripts -- very similar
   to UDFs [1]
   2. Users can compile inline Groovy UDFs and use them in queries [2]
   3. Users can dynamically add/delete jars, or even execute arbitrary
   shell command [3]

These features cannot be supported merely by a parser/planner, and it's
open to discussion whether Flink even should support them at all.

So the ultimate goal of this FLIP is to provide Hive syntax compatibility
to features that are already available in Flink, which I believe will cover
most common use cases.

[1]
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-TRANSFORMExamples
[2]
https://community.cloudera.com/t5/Community-Articles/Apache-Hive-Groovy-UDF-examples/ta-p/245060
[3]
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveInteractiveShellCommands

On Thu, Dec 10, 2020 at 6:11 PM Zhijiang 
wrote:

> Thanks for launching the discussion and the FLIP, Rui!
>
> It is really nice to see our continuous efforts for compatibility with
> Hive and benefiting users in this area.
> I am only curious that are there any other compatible limitations for Hive
> users after this FLIP? Or can I say that the Hive compatibility is
> completely resolved after this FLIP?
> I am interested in the ultimate goal in this area. Maybe it is out of this
> FLIP scope, but still wish some insights from you if possible. :)
>
> Best,
> Zhijiang
>
>
> --
> From:Rui Li 
> Send Time:2020年12月10日(星期四) 16:46
> To:dev 
> Subject:Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility
>
> Thanks Kurt for your inputs!
>
> I agree we should extend Hive code to support non-Hive tables. I have
> updated the wiki page to remove the limitations you mentioned, and add
> typical use cases in the "Motivation" section.
>
> Regarding comment #b, the interface is defined in flink-table-planner-blink
> and only used by the blink planner. So I think "BlinkParserFactory" is a
> better name, WDYT?
>
> On Mon, Dec 7, 2020 at 12:28 PM Kurt Young  wrote:
>
> > Thanks Rui for starting this discussion.
> >
> > I can see the benefit that we improve hive compatibility further, as
> quite
> > some users are asking for this
> > feature in mailing lists [1][2][3] and some online chatting tools such as
> > DingTalk.
> >
> > I have 3 comments regarding to the design doc:
> >
> > a) Could you add a section to describe the typical use case you want to
> > support after this feature is introduced?
> > In that way, users can also have an impression how to use this feature
> and
> > what the behavior and outcome will be.
> >
> > b) Regarding the naming: "BlinkParserFactory", I suggest renaming it to
> > "FlinkParserFactory".
> >
> > c) About the two limitations you mentioned:
> > 1. Only works with Hive tables and the current catalog needs to be a
> > HiveCatalog.
> > 2. Queries cannot involve tables/views from multiple catalogs.
> > I assume this is because hive parser and analyzer doesn't support
> > referring to a name with "x.y.z" fashion? Since
> > we can control all the behaviors by leveraging the codes hive currently
> > use. Is it possible that we can remove such
> > limitations? The reason is I'm not sure if users can make the whole story
> > work purely depending on hive catalog (that's
> > the reason why I gave comment #a). If multiple catalogs are involved,
> with
> > this limitation I don't think any meaningful
> > pipeline could be built. For example, users want to stream data from
> Kafka
> > to Hive, fully use hive's dialect including
> > query part. The kafka table could be a temporary table or saved in
> default
> > memory catalog.
> >
> >
> > [1] http://apache-flink.147419.n8.nabble.com/calcite-td9059.html#a9118
> > [2]
> http://apache-flink.147419.n8.nabble.com/hive-sql-flink-11-td9116.html
> > [3]
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-to-in-Flink-to-support-below-HIVE-SQL-td34162.html
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Dec 2, 2020 at 10:02 PM Rui Li  wrote:
> >
> > > Hi guys,
> > >
> > > I'd like to start a discussion about providing HiveQL compatibility for
> > > users connecting to a hive warehouse. FLIP-123 has already covered most
> > > DDLs. So now it's time to complement the other big missing part --
> > queries.
> > > With FLIP-152, the hive dialect covers more scenarios and makes 

[jira] [Created] (FLINK-20569) testKafkaSourceSinkWithMetadata hangs

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-20569:


 Summary: testKafkaSourceSinkWithMetadata hangs
 Key: FLINK-20569
 URL: https://issues.apache.org/jira/browse/FLINK-20569
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Table SQL / Ecosystem
Affects Versions: 1.12.0, 1.13.0
Reporter: Huang Xingbo


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10781=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=f266c805-9429-58ed-2f9e-482e7b82f58b]
{code:java}
2020-12-10T23:10:46.7788275Z Test testKafkaSourceSinkWithMetadata[legacy = 
false, format = 
csv](org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase) is 
running.
2020-12-10T23:10:46.7789360Z 

2020-12-10T23:10:46.7790602Z 23:10:46,776 [main] INFO  
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] - 
Creating topic metadata_topic_csv
2020-12-10T23:10:47.1145296Z 23:10:47,112 [main] WARN  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property 
[transaction.timeout.ms] not specified. Setting it to 360 ms
2020-12-10T23:10:47.1683896Z 23:10:47,166 [Sink: 
Sink(table=[default_catalog.default_database.kafka], fields=[physical_1, 
physical_2, physical_3, headers, timestamp]) (1/1)#0] WARN  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Using 
AT_LEAST_ONCE semantic, but checkpointing is not enabled. Switching to NONE 
semantic.
2020-12-10T23:10:47.2087733Z 23:10:47,206 [Sink: 
Sink(table=[default_catalog.default_database.kafka], fields=[physical_1, 
physical_2, physical_3, headers, timestamp]) (1/1)#0] INFO  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Starting 
FlinkKafkaInternalProducer (1/1) to produce into default topic 
metadata_topic_csv
2020-12-10T23:10:47.5157133Z 23:10:47,513 [Source: 
TableSourceScan(table=[[default_catalog, default_database, kafka]], 
fields=[physical_1, physical_2, physical_3, topic, partition, headers, 
leader-epoch, timestamp, timestamp-type]) -> Calc(select=[physical_1, 
physical_2, CAST(timestamp-type) AS timestamp-type, CAST(timestamp) AS 
timestamp, leader-epoch, CAST(headers) AS headers, CAST(partition) AS 
partition, CAST(topic) AS topic, physical_3]) -> SinkConversionToTuple2 -> 
Sink: Select table sink (1/1)#0] INFO  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] - 
Consumer subtask 0 has no restore state.
2020-12-10T23:10:47.5233388Z 23:10:47,521 [Source: 
TableSourceScan(table=[[default_catalog, default_database, kafka]], 
fields=[physical_1, physical_2, physical_3, topic, partition, headers, 
leader-epoch, timestamp, timestamp-type]) -> Calc(select=[physical_1, 
physical_2, CAST(timestamp-type) AS timestamp-type, CAST(timestamp) AS 
timestamp, leader-epoch, CAST(headers) AS headers, CAST(partition) AS 
partition, CAST(topic) AS topic, physical_3]) -> SinkConversionToTuple2 -> 
Sink: Select table sink (1/1)#0] INFO  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] - 
Consumer subtask 0 will start reading the following 1 partitions from the 
earliest offsets: [KafkaTopicPartition{topic='metadata_topic_csv', partition=0}]
2020-12-10T23:10:47.5387239Z 23:10:47,537 [Legacy Source Thread - Source: 
TableSourceScan(table=[[default_catalog, default_database, kafka]], 
fields=[physical_1, physical_2, physical_3, topic, partition, headers, 
leader-epoch, timestamp, timestamp-type]) -> Calc(select=[physical_1, 
physical_2, CAST(timestamp-type) AS timestamp-type, CAST(timestamp) AS 
timestamp, leader-epoch, CAST(headers) AS headers, CAST(partition) AS 
partition, CAST(topic) AS topic, physical_3]) -> SinkConversionToTuple2 -> 
Sink: Select table sink (1/1)#0] INFO  
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] - 
Consumer subtask 0 creating fetcher with offsets 
{KafkaTopicPartition{topic='metadata_topic_csv', partition=0}=-915623761775}.
2020-12-11T02:34:02.6860452Z ##[error]The operation was canceled.
{code}
This test started at 2020-12-10T23:10:46.7788275Z and has not been finished at 
2020-12-11T02:34:02.6860452Z



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.11.3, release candidate #1

[Xintong Song]

Hi everyone,

Stephan and Becket reported some missing commits from the FLIP-27
backportings, some of which are quite significant that they would consider
as release blockers.

Therefore, I'm hereby canceling this release candidate. The next release
candidate will be created as soon as the missing changes are backported.

Thank you~

Xintong Song



On Thu, Dec 10, 2020 at 5:31 PM Xintong Song  wrote:

> Hi Yun,
>
> Thanks for the information.
>
> IMHO, FLINK-20554 should not be a release blocker. I'd rather not reject
> RC1 on it.
> However, if the RC1 does not pass this vote, and if the issue is managed
> to be fixed before the next RC, we can still include it for release 1.11.3.
>
> BTW, I think there only existed 'release-1.11.3-rc1' tag in current code
>> base instead of your descripted 'release-1.11.3-rc3' in previous mail
>> thread.
>>
> Sorry for the typo. It should be 'release-1.11.3-rc1'.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Dec 10, 2020 at 4:16 PM Yun Tang  wrote:
>
>> Hi Xintong
>>
>> Thanks for driving this release! Someone reported that the checkpointed
>> data size on overview page is broken [1] and I confirmed that really
>> exists. I hope to include that fix in release-1.11.3 as that could be a
>> quick fix and impact the usability.
>>
>> BTW, I think there only existed 'release-1.11.3-rc1' tag in current code
>> base instead of your descripted 'release-1.11.3-rc3' in previous mail
>> thread.
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20554
>>
>> Best
>> Yun Tang
>> 
>> From: Xintong Song 
>> Sent: Wednesday, December 9, 2020 11:14
>> To: dev 
>> Cc: tzuli...@apache.org 
>> Subject: [VOTE] Release 1.11.3, release candidate #1
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version 1.11.3,
>> as follows:
>>
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.11.3-rc3" [5],
>> * website pull request listing the new release and adding announcement
>> blog
>> post [6].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Xintong Song
>>
>> [1]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348761
>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.3-rc1
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1405
>>
>> https://repository.apache.org/content/repositories/orgapacheflink-1406
>> [5] https://github.com/apache/flink/releases/tag/release-1.11.3-rc1
>> [6] https://github.com/apache/flink-web/pull/399
>>
>

[jira] [Created] (FLINK-20568) Kerberized YARN per-job on Docker test failed with "Hadoop security with Kerberos is enabled but the login user does not have Kerberos credentials or delegation tokens!"

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-20568:


 Summary: Kerberized YARN per-job on Docker test failed with 
"Hadoop security with Kerberos is enabled but the login user does not have 
Kerberos credentials or delegation tokens!"
 Key: FLINK-20568
 URL: https://issues.apache.org/jira/browse/FLINK-20568
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.12.0, 1.11.0, 1.13.0
Reporter: Huang Xingbo


Instance on 1.11 branch

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10780=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994]
{code:java}
2020-12-10T22:38:25.1087443Z  The program finished with the following exception:
2020-12-10T22:38:25.1087688Z 
2020-12-10T22:38:25.1088094Z 
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: Could not deploy Yarn job cluster.
2020-12-10T22:38:25.1088717Zat 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
2020-12-10T22:38:25.1089321Zat 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
2020-12-10T22:38:25.1090233Zat 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
2020-12-10T22:38:25.1090749Zat 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
2020-12-10T22:38:25.1091233Zat 
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
2020-12-10T22:38:25.1091705Zat 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
2020-12-10T22:38:25.1092225Zat 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
2020-12-10T22:38:25.1095464Zat 
java.security.AccessController.doPrivileged(Native Method)
2020-12-10T22:38:25.1095961Zat 
javax.security.auth.Subject.doAs(Subject.java:422)
2020-12-10T22:38:25.1096436Zat 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
2020-12-10T22:38:25.1097027Zat 
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
2020-12-10T22:38:25.1097859Zat 
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
2020-12-10T22:38:25.1098474Z Caused by: 
org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy 
Yarn job cluster.
2020-12-10T22:38:25.1099065Zat 
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:431)
2020-12-10T22:38:25.1099674Zat 
org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70)
2020-12-10T22:38:25.1100918Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1818)
2020-12-10T22:38:25.1101607Zat 
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128)
2020-12-10T22:38:25.1102202Zat 
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
2020-12-10T22:38:25.1102840Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700)
2020-12-10T22:38:25.1103467Zat 
org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:96)
2020-12-10T22:38:25.1104174Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-12-10T22:38:25.1104638Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-12-10T22:38:25.1105174Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-12-10T22:38:25.1105645Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-12-10T22:38:25.1106119Zat 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
2020-12-10T22:38:25.1106495Z... 11 more
2020-12-10T22:38:25.1106940Z Caused by: java.lang.RuntimeException: Hadoop 
security with Kerberos is enabled but the login user does not have Kerberos 
credentials or delegation tokens!
2020-12-10T22:38:25.1107584Zat 
org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:475)
2020-12-10T22:38:25.1108484Zat 
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:424)
2020-12-10T22:38:25.1109201Z... 22 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20567) Document Error

[appleyuchi (Jira)]

appleyuchi created FLINK-20567:
--

 Summary: Document Error
 Key: FLINK-20567
 URL: https://issues.apache.org/jira/browse/FLINK-20567
 Project: Flink
  Issue Type: Test
Reporter: appleyuchi



||item||Content||
|Document|[Link|https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/tableApi.html]|
|part|Inner Join with Table Function (UDTF)|
|origin|TableFunction split = new MySplitUDTF();|
|change to|TableFunction> split = new 
MySplitUDTF();|


I have run the following the codes successfully 
that contain all the contents from the above.
①[InnerJoinwithTableFunction.java|https://paste.ubuntu.com/p/MMXJPrfRWC]
②[MySplitUDTF.java|https://paste.ubuntu.com/p/Q6fDHxw4Td/]











--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[nick toker]

Hi
first good job  and tank you

i don't find in docker hub the new version 1.12

when it will be there ?

nick

‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
rmetz...@apache.org‬‏>:‬

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.12.0, which is the latest major release.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Dian & Robert
>
>

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Robert Metzger]

Thanks Nick!

This is blocked on the maintainers of the official images to merge this PR:
https://github.com/docker-library/official-images/pull/9249
I can not really make an estimate when they'll merge it.

On Thu, Dec 10, 2020 at 4:20 PM nick toker  wrote:

> Hi
> first good job  and tank you
>
> i don't find in docker hub the new version 1.12
>
> when it will be there ?
>
> nick
>
> ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> rmetz...@apache.org‬‏>:‬
>
>> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.12.0, which is the latest major release.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the
>> improvements for this bugfix release:
>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
>>
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>>
>> Regards,
>> Dian & Robert
>>
>>

Re: Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Till Rohrmann]

Congratulations to everyone who contributed and thanks a lot for being our
release managers Dian and Robert!

Cheers,
Till

On Thu, Dec 10, 2020 at 1:47 PM Rui Li  wrote:

> Congrats! Thanks to Dian and Robert for driving the release!
>
> On Thu, Dec 10, 2020 at 8:42 PM hailongwang <18868816...@163.com> wrote:
>
> >
> >
> > Congratulations! Thanks Dian and Robert for the great work!
> >
> >
> > Best,
> > Hailong
> > At 2020-12-10 19:35:44, "Yun Tang"  wrote:
> > >Thanks Dian and Robert for driving this release and thanks everyone who
> > makes this great work possible !
> > >
> > >Best
> > >Yun Tang
> > >
> > >From: Wei Zhong 
> > >Sent: Thursday, December 10, 2020 20:32
> > >To: dev@flink.apache.org 
> > >Cc: user ; annou...@apache.org <
> > annou...@apache.org>
> > >Subject: Re: [ANNOUNCE] Apache Flink 1.12.0 released
> > >
> > >Congratulations! Thanks Dian and Robert for the great work!
> > >
> > >Best,
> > >Wei
> > >
> > >> 在 2020年12月10日，20:26，Leonard Xu  写道：
> > >>
> > >>
> > >> Thanks Dian and Robert for the great work as release manager !
> > >> And thanks everyone who makes the release possible !
> > >>
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >>> 在 2020年12月10日，20:17，Robert Metzger  写道：
> > >>>
> > >>> The Apache Flink community is very happy to announce the release of
> > Apache
> > >>> Flink 1.12.0, which is the latest major release.
> > >>>
> > >>> Apache Flink® is an open-source stream processing framework for
> > >>> distributed, high-performing, always-available, and accurate data
> > streaming
> > >>> applications.
> > >>>
> > >>> The release is available for download at:
> > >>> https://flink.apache.org/downloads.html
> > >>>
> > >>> Please check out the release blog post for an overview of the
> > improvements
> > >>> for this bugfix release:
> > >>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> > >>>
> > >>> The full release notes are available in Jira:
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> > >>>
> > >>> We would like to thank all contributors of the Apache Flink community
> > who
> > >>> made this release possible!
> > >>>
> > >>> Regards,
> > >>> Dian & Robert
> > >>
> > >
> >
>
>
> --
> Best regards!
> Rui Li
>

[jira] [Created] (FLINK-20566) Fix typo in `Cassandra Connector` docs.

[xiaozilong (Jira)]

xiaozilong created FLINK-20566:
--

 Summary: Fix typo in `Cassandra Connector` docs.
 Key: FLINK-20566
 URL: https://issues.apache.org/jira/browse/FLINK-20566
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.12.0
Reporter: xiaozilong


Fix typo in [Cassandra 
Connector|https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/cassandra.html]
 docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Rui Li]

Congrats! Thanks to Dian and Robert for driving the release!

On Thu, Dec 10, 2020 at 8:42 PM hailongwang <18868816...@163.com> wrote:

>
>
> Congratulations! Thanks Dian and Robert for the great work!
>
>
> Best,
> Hailong
> At 2020-12-10 19:35:44, "Yun Tang"  wrote:
> >Thanks Dian and Robert for driving this release and thanks everyone who
> makes this great work possible !
> >
> >Best
> >Yun Tang
> >
> >From: Wei Zhong 
> >Sent: Thursday, December 10, 2020 20:32
> >To: dev@flink.apache.org 
> >Cc: user ; annou...@apache.org <
> annou...@apache.org>
> >Subject: Re: [ANNOUNCE] Apache Flink 1.12.0 released
> >
> >Congratulations! Thanks Dian and Robert for the great work!
> >
> >Best,
> >Wei
> >
> >> 在 2020年12月10日，20:26，Leonard Xu  写道：
> >>
> >>
> >> Thanks Dian and Robert for the great work as release manager !
> >> And thanks everyone who makes the release possible !
> >>
> >>
> >> Best,
> >> Leonard
> >>
> >>> 在 2020年12月10日，20:17，Robert Metzger  写道：
> >>>
> >>> The Apache Flink community is very happy to announce the release of
> Apache
> >>> Flink 1.12.0, which is the latest major release.
> >>>
> >>> Apache Flink® is an open-source stream processing framework for
> >>> distributed, high-performing, always-available, and accurate data
> streaming
> >>> applications.
> >>>
> >>> The release is available for download at:
> >>> https://flink.apache.org/downloads.html
> >>>
> >>> Please check out the release blog post for an overview of the
> improvements
> >>> for this bugfix release:
> >>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> >>>
> >>> The full release notes are available in Jira:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> >>>
> >>> We would like to thank all contributors of the Apache Flink community
> who
> >>> made this release possible!
> >>>
> >>> Regards,
> >>> Dian & Robert
> >>
> >
>


-- 
Best regards!
Rui Li

Re: Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Xingbo Huang]

Congratulations! Thanks Dian and Robert for the great work! And thanks
everyone who makes the release possible !

Best,
Xingbo

hailongwang <18868816...@163.com> 于2020年12月10日周四 下午8:42写道：

>
>
> Congratulations! Thanks Dian and Robert for the great work!
>
>
> Best,
> Hailong
> At 2020-12-10 19:35:44, "Yun Tang"  wrote:
> >Thanks Dian and Robert for driving this release and thanks everyone who
> makes this great work possible !
> >
> >Best
> >Yun Tang
> >
> >From: Wei Zhong 
> >Sent: Thursday, December 10, 2020 20:32
> >To: dev@flink.apache.org 
> >Cc: user ; annou...@apache.org <
> annou...@apache.org>
> >Subject: Re: [ANNOUNCE] Apache Flink 1.12.0 released
> >
> >Congratulations! Thanks Dian and Robert for the great work!
> >
> >Best,
> >Wei
> >
> >> 在 2020年12月10日，20:26，Leonard Xu  写道：
> >>
> >>
> >> Thanks Dian and Robert for the great work as release manager !
> >> And thanks everyone who makes the release possible !
> >>
> >>
> >> Best,
> >> Leonard
> >>
> >>> 在 2020年12月10日，20:17，Robert Metzger  写道：
> >>>
> >>> The Apache Flink community is very happy to announce the release of
> Apache
> >>> Flink 1.12.0, which is the latest major release.
> >>>
> >>> Apache Flink® is an open-source stream processing framework for
> >>> distributed, high-performing, always-available, and accurate data
> streaming
> >>> applications.
> >>>
> >>> The release is available for download at:
> >>> https://flink.apache.org/downloads.html
> >>>
> >>> Please check out the release blog post for an overview of the
> improvements
> >>> for this bugfix release:
> >>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> >>>
> >>> The full release notes are available in Jira:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> >>>
> >>> We would like to thank all contributors of the Apache Flink community
> who
> >>> made this release possible!
> >>>
> >>> Regards,
> >>> Dian & Robert
> >>
> >
>

Re:Re: [ANNOUNCE] Apache Flink 1.12.0 released

[hailongwang]

Congratulations! Thanks Dian and Robert for the great work!


Best,
Hailong
At 2020-12-10 19:35:44, "Yun Tang"  wrote:
>Thanks Dian and Robert for driving this release and thanks everyone who makes 
>this great work possible !
>
>Best
>Yun Tang
>
>From: Wei Zhong 
>Sent: Thursday, December 10, 2020 20:32
>To: dev@flink.apache.org 
>Cc: user ; annou...@apache.org 
>Subject: Re: [ANNOUNCE] Apache Flink 1.12.0 released
>
>Congratulations! Thanks Dian and Robert for the great work!
>
>Best,
>Wei
>
>> 在 2020年12月10日，20:26，Leonard Xu  写道：
>>
>>
>> Thanks Dian and Robert for the great work as release manager !
>> And thanks everyone who makes the release possible !
>>
>>
>> Best,
>> Leonard
>>
>>> 在 2020年12月10日，20:17，Robert Metzger  写道：
>>>
>>> The Apache Flink community is very happy to announce the release of Apache
>>> Flink 1.12.0, which is the latest major release.
>>>
>>> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the improvements
>>> for this bugfix release:
>>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
>>>
>>> The full release notes are available in Jira:
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
>>>
>>> We would like to thank all contributors of the Apache Flink community who
>>> made this release possible!
>>>
>>> Regards,
>>> Dian & Robert
>>
>

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Yun Tang]

Thanks Dian and Robert for driving this release and thanks everyone who makes 
this great work possible !

Best
Yun Tang

From: Wei Zhong 
Sent: Thursday, December 10, 2020 20:32
To: dev@flink.apache.org 
Cc: user ; annou...@apache.org 
Subject: Re: [ANNOUNCE] Apache Flink 1.12.0 released

Congratulations! Thanks Dian and Robert for the great work!

Best,
Wei

> 在 2020年12月10日，20:26，Leonard Xu  写道：
>
>
> Thanks Dian and Robert for the great work as release manager !
> And thanks everyone who makes the release possible !
>
>
> Best,
> Leonard
>
>> 在 2020年12月10日，20:17，Robert Metzger  写道：
>>
>> The Apache Flink community is very happy to announce the release of Apache
>> Flink 1.12.0, which is the latest major release.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the improvements
>> for this bugfix release:
>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
>>
>> The full release notes are available in Jira:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
>>
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>>
>> Regards,
>> Dian & Robert
>

Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

[Rui Li]

Hi Zhijiang,

Glad to know you're interested in this FLIP. I wouldn't claim 100%
compatibility with this FLIP. That's because Flink doesn't have the
functionalities to support all Hive's features. To list a few examples:

   1. Hive allows users to process data with shell scripts -- very similar
   to UDFs [1]
   2. Users can compile inline Groovy UDFs and use them in queries [2]
   3. Users can dynamically add/delete jars, or even execute arbitrary
   shell command [3]

These features cannot be supported merely by a parser/planner, and it's
open to discussion whether Flink even should support them at all.

So the ultimate goal of this FLIP is to provide Hive syntax compatibility
to features that are already available in Flink, which I believe will cover
most common use cases.

[1]
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-TRANSFORMExamples
[2]
https://community.cloudera.com/t5/Community-Articles/Apache-Hive-Groovy-UDF-examples/ta-p/245060
[3]
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveInteractiveShellCommands

On Thu, Dec 10, 2020 at 6:11 PM Zhijiang 
wrote:

> Thanks for launching the discussion and the FLIP, Rui!
>
> It is really nice to see our continuous efforts for compatibility with
> Hive and benefiting users in this area.
> I am only curious that are there any other compatible limitations for Hive
> users after this FLIP? Or can I say that the Hive compatibility is
> completely resolved after this FLIP?
> I am interested in the ultimate goal in this area. Maybe it is out of this
> FLIP scope, but still wish some insights from you if possible. :)
>
> Best,
> Zhijiang
>
>
> --
> From:Rui Li 
> Send Time:2020年12月10日(星期四) 16:46
> To:dev 
> Subject:Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility
>
> Thanks Kurt for your inputs!
>
> I agree we should extend Hive code to support non-Hive tables. I have
> updated the wiki page to remove the limitations you mentioned, and add
> typical use cases in the "Motivation" section.
>
> Regarding comment #b, the interface is defined in flink-table-planner-blink
> and only used by the blink planner. So I think "BlinkParserFactory" is a
> better name, WDYT?
>
> On Mon, Dec 7, 2020 at 12:28 PM Kurt Young  wrote:
>
> > Thanks Rui for starting this discussion.
> >
> > I can see the benefit that we improve hive compatibility further, as
> quite
> > some users are asking for this
> > feature in mailing lists [1][2][3] and some online chatting tools such as
> > DingTalk.
> >
> > I have 3 comments regarding to the design doc:
> >
> > a) Could you add a section to describe the typical use case you want to
> > support after this feature is introduced?
> > In that way, users can also have an impression how to use this feature
> and
> > what the behavior and outcome will be.
> >
> > b) Regarding the naming: "BlinkParserFactory", I suggest renaming it to
> > "FlinkParserFactory".
> >
> > c) About the two limitations you mentioned:
> > 1. Only works with Hive tables and the current catalog needs to be a
> > HiveCatalog.
> > 2. Queries cannot involve tables/views from multiple catalogs.
> > I assume this is because hive parser and analyzer doesn't support
> > referring to a name with "x.y.z" fashion? Since
> > we can control all the behaviors by leveraging the codes hive currently
> > use. Is it possible that we can remove such
> > limitations? The reason is I'm not sure if users can make the whole story
> > work purely depending on hive catalog (that's
> > the reason why I gave comment #a). If multiple catalogs are involved,
> with
> > this limitation I don't think any meaningful
> > pipeline could be built. For example, users want to stream data from
> Kafka
> > to Hive, fully use hive's dialect including
> > query part. The kafka table could be a temporary table or saved in
> default
> > memory catalog.
> >
> >
> > [1] http://apache-flink.147419.n8.nabble.com/calcite-td9059.html#a9118
> > [2]
> http://apache-flink.147419.n8.nabble.com/hive-sql-flink-11-td9116.html
> > [3]
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-to-in-Flink-to-support-below-HIVE-SQL-td34162.html
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Dec 2, 2020 at 10:02 PM Rui Li  wrote:
> >
> > > Hi guys,
> > >
> > > I'd like to start a discussion about providing HiveQL compatibility for
> > > users connecting to a hive warehouse. FLIP-123 has already covered most
> > > DDLs. So now it's time to complement the other big missing part --
> > queries.
> > > With FLIP-152, the hive dialect covers more scenarios and makes it even
> > > easier for users to migrate to Flink. More details are in the FLIP wiki
> > > page [1]. Looking forward to your feedback!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-152%3A+Hive+Query+Syntax+Compatibility
> > >
> > > --
> > > 

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Wei Zhong]

Congratulations! Thanks Dian and Robert for the great work!

Best,
Wei

> 在 2020年12月10日，20:26，Leonard Xu  写道：
> 
> 
> Thanks Dian and Robert for the great work as release manager ! 
> And thanks everyone who makes the release possible ! 
> 
> 
> Best,
> Leonard
> 
>> 在 2020年12月10日，20:17，Robert Metzger  写道：
>> 
>> The Apache Flink community is very happy to announce the release of Apache
>> Flink 1.12.0, which is the latest major release.
>> 
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>> 
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>> 
>> Please check out the release blog post for an overview of the improvements
>> for this bugfix release:
>> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
>> 
>> The full release notes are available in Jira:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
>> 
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>> 
>> Regards,
>> Dian & Robert
> 

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Leonard Xu]

Thanks Dian and Robert for the great work as release manager ! 
 And thanks everyone who makes the release possible ! 


Best,
Leonard

> 在 2020年12月10日，20:17，Robert Metzger  写道：
> 
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.12.0, which is the latest major release.
> 
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
> 
> The release is available for download at:
> https://flink.apache.org/downloads.html
> 
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> 
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
> 
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
> 
> Regards,
> Dian & Robert

[ANNOUNCE] Apache Flink 1.12.0 released

[Robert Metzger]

The Apache Flink community is very happy to announce the release of Apache
Flink 1.12.0, which is the latest major release.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements
for this bugfix release:
https://flink.apache.org/news/2020/12/10/release-1.12.0.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Dian & Robert

[jira] [Created] (FLINK-20565) Fix typo in EXPLAIN Statements docs.

[xiaozilong (Jira)]

xiaozilong created FLINK-20565:
--

 Summary: Fix typo in EXPLAIN Statements docs.
 Key: FLINK-20565
 URL: https://issues.apache.org/jira/browse/FLINK-20565
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.12.0, 1.11.0
Reporter: xiaozilong


Fix typo in EXPLAIN Statements docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20564) Add metrics for ElasticSearch connector

[Peidian Li (Jira)]

Peidian Li created FLINK-20564:
--

 Summary: Add metrics for ElasticSearch connector 
 Key: FLINK-20564
 URL: https://issues.apache.org/jira/browse/FLINK-20564
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / ElasticSearch
Affects Versions: 1.12.0
Reporter: Peidian Li


The current ElasticSearch connector lacks some metric, could we add some 
metrics such as P95、P99、and the failed number of BulkRequest.
We can implement it in the 
[BulkProcessorListener|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L389]
 callback function.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20563) Support built-in functions for Hive versions prior to 1.2.0

[Rui Li (Jira)]

Rui Li created FLINK-20563:
--

 Summary: Support built-in functions for Hive versions prior to 
1.2.0
 Key: FLINK-20563
 URL: https://issues.apache.org/jira/browse/FLINK-20563
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Reporter: Rui Li


Currently Hive built-in functions are supported only for Hive-1.2.0 and later. 
We should investigate how to lift this limitation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

[Zhijiang]

Thanks for launching the discussion and the FLIP, Rui!

It is really nice to see our continuous efforts for compatibility with Hive and 
benefiting users in this area.
I am only curious that are there any other compatible limitations for Hive 
users after this FLIP? Or can I say that the Hive compatibility is completely 
resolved after this FLIP?
I am interested in the ultimate goal in this area. Maybe it is out of this FLIP 
scope, but still wish some insights from you if possible. :)

Best,
Zhijiang


--
From:Rui Li 
Send Time:2020年12月10日(星期四) 16:46
To:dev 
Subject:Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

Thanks Kurt for your inputs!

I agree we should extend Hive code to support non-Hive tables. I have
updated the wiki page to remove the limitations you mentioned, and add
typical use cases in the "Motivation" section.

Regarding comment #b, the interface is defined in flink-table-planner-blink
and only used by the blink planner. So I think "BlinkParserFactory" is a
better name, WDYT?

On Mon, Dec 7, 2020 at 12:28 PM Kurt Young  wrote:

> Thanks Rui for starting this discussion.
>
> I can see the benefit that we improve hive compatibility further, as quite
> some users are asking for this
> feature in mailing lists [1][2][3] and some online chatting tools such as
> DingTalk.
>
> I have 3 comments regarding to the design doc:
>
> a) Could you add a section to describe the typical use case you want to
> support after this feature is introduced?
> In that way, users can also have an impression how to use this feature and
> what the behavior and outcome will be.
>
> b) Regarding the naming: "BlinkParserFactory", I suggest renaming it to
> "FlinkParserFactory".
>
> c) About the two limitations you mentioned:
> 1. Only works with Hive tables and the current catalog needs to be a
> HiveCatalog.
> 2. Queries cannot involve tables/views from multiple catalogs.
> I assume this is because hive parser and analyzer doesn't support
> referring to a name with "x.y.z" fashion? Since
> we can control all the behaviors by leveraging the codes hive currently
> use. Is it possible that we can remove such
> limitations? The reason is I'm not sure if users can make the whole story
> work purely depending on hive catalog (that's
> the reason why I gave comment #a). If multiple catalogs are involved, with
> this limitation I don't think any meaningful
> pipeline could be built. For example, users want to stream data from Kafka
> to Hive, fully use hive's dialect including
> query part. The kafka table could be a temporary table or saved in default
> memory catalog.
>
>
> [1] http://apache-flink.147419.n8.nabble.com/calcite-td9059.html#a9118
> [2] http://apache-flink.147419.n8.nabble.com/hive-sql-flink-11-td9116.html
> [3]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-to-in-Flink-to-support-below-HIVE-SQL-td34162.html
>
> Best,
> Kurt
>
>
> On Wed, Dec 2, 2020 at 10:02 PM Rui Li  wrote:
>
> > Hi guys,
> >
> > I'd like to start a discussion about providing HiveQL compatibility for
> > users connecting to a hive warehouse. FLIP-123 has already covered most
> > DDLs. So now it's time to complement the other big missing part --
> queries.
> > With FLIP-152, the hive dialect covers more scenarios and makes it even
> > easier for users to migrate to Flink. More details are in the FLIP wiki
> > page [1]. Looking forward to your feedback!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-152%3A+Hive+Query+Syntax+Compatibility
> >
> > --
> > Best regards!
> > Rui Li
> >
>


-- 
Best regards!
Rui Li

[jira] [Created] (FLINK-20562) Support ExplainDetails for EXPLAIN sytnax

[Jark Wu (Jira)]

Jark Wu created FLINK-20562:
---

 Summary: Support ExplainDetails for EXPLAIN sytnax
 Key: FLINK-20562
 URL: https://issues.apache.org/jira/browse/FLINK-20562
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: Jark Wu
 Fix For: 1.13.0


Currently, {{EXPLAIN}} syntax only supports to print the default AST, logical 
plan, and physical plan. However, it doesn't support to print detailed 
information such as CHANGELOG_MODE, ESTIMATED_COST, JSON_EXECUTION_PLAN which 
are defined in {{ExplainDetail}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Pod template for native K8s integration

[Till Rohrmann]

Hi Yang Wang,

thanks for starting this discussion. I like the idea of providing a general
mechanism to customize Flink's K8s deployment instead of implementing each
feature separately. This should lower the maintenance overhead and people
coming from K8s should already be familiar with how to specify the pod
template.

Cheers,
Till

On Wed, Dec 9, 2020 at 4:44 PM Yang Wang  wrote:

> Hi devs,
>
> So far we keep developing the native K8s integration for three major
> release cycles in Flink.
>
>-
>
>Flink 1.10
>-
>
>   Beta version
>   -
>
>   Dynamic resource allocation
>   -
>
>   Session mode only
>   -
>
>Flink 1.11
>-
>
>   Application mode
>   -
>
>   Usability Improvements
>   -
>
>   GPU resource support
>   -
>
>Flink 1.12
>-
>
>   Security improvements(kerberos, K8s secret)
>   -
>
>   PyFlink support
>   -
>
>   K8s HA
>
> Now the native K8s integration is almost production ready. We already have
> the common features for deploying a Flink cluster on K8s cluster natively.
> However, compared with the current supported features[1] in
> flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am
> afraid that we still need some advanced features.
>
>-
>
>Init containers[2]
>-
>
>Sidecar containers[3]
>-
>
>Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4]
>-
>
>Set owner reference for the JobManager deployment[5]
>-
>
>Support customizing of containers for native kubernetes setup[6]
>-
>
>Pod Affinity and Anti-affinity
>-
>
>Pod Security Context
>-
>
>Pod Environment Variables Reference
>
> Although they may not be used very frequently or we already have some work
> around, I think they are still important for the production. So I am
> considering whether we could introduce a pod template[7] for all the use
> cases. Users could configure a local yaml file for JobManager/TaskManager.
> This will let Flink create the JobManager/TaskManager pods with the
> template first, not the empty. Of course, the Flink config options for
> Kubernetes[8] have high priority and could overwrite these values. The pod
> template could help us get rid of maintaining more and more K8s features.
>
>
> Looking forward to your suggestions.
>
>
> [1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features
>
> [2]. https://issues.apache.org/jira/browse/FLINK-15641
>
> [3]. https://issues.apache.org/jira/browse/FLINK-15871
>
> [4]. https://issues.apache.org/jira/browse/FLINK-15649
>
> [5]. https://issues.apache.org/jira/browse/FLINK-20359
>
> [6]. https://issues.apache.org/jira/browse/FLINK-20324
>
> [7]. https://issues.apache.org/jira/browse/FLINK-15656
>
> [8].
>
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes
>
>
> Best,
>
> Yang
>

Re: [VOTE] Release 1.11.3, release candidate #1

[Xintong Song]

Hi Yun,

Thanks for the information.

IMHO, FLINK-20554 should not be a release blocker. I'd rather not reject
RC1 on it.
However, if the RC1 does not pass this vote, and if the issue is managed to
be fixed before the next RC, we can still include it for release 1.11.3.

BTW, I think there only existed 'release-1.11.3-rc1' tag in current code
> base instead of your descripted 'release-1.11.3-rc3' in previous mail
> thread.
>
Sorry for the typo. It should be 'release-1.11.3-rc1'.

Thank you~

Xintong Song



On Thu, Dec 10, 2020 at 4:16 PM Yun Tang  wrote:

> Hi Xintong
>
> Thanks for driving this release! Someone reported that the checkpointed
> data size on overview page is broken [1] and I confirmed that really
> exists. I hope to include that fix in release-1.11.3 as that could be a
> quick fix and impact the usability.
>
> BTW, I think there only existed 'release-1.11.3-rc1' tag in current code
> base instead of your descripted 'release-1.11.3-rc3' in previous mail
> thread.
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-20554
>
> Best
> Yun Tang
> 
> From: Xintong Song 
> Sent: Wednesday, December 9, 2020 11:14
> To: dev 
> Cc: tzuli...@apache.org 
> Subject: [VOTE] Release 1.11.3, release candidate #1
>
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.11.3,
> as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.11.3-rc3" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Xintong Song
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348761
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.3-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1405
>
> https://repository.apache.org/content/repositories/orgapacheflink-1406
> [5] https://github.com/apache/flink/releases/tag/release-1.11.3-rc1
> [6] https://github.com/apache/flink-web/pull/399
>

Re: [DISCUSS] Programmatically submit Flink job jar to session cluster

[Flavio Pompermaier]

To me creating the PackagedProgram on the client side is very bad, at least
for 2 things:
   1. You must ensure to have almost the same classpath of the Flink
cluster otherwise you can face problems in deserializing the submitted job
graph (for example jackson automatically tries to create modules that can
be found on the client classpath if using spring but not on the job
manager...that's exactly what happened to me initially)
   2. Also if you manage to create the PackagedProgram correctly, Job
listeners are not fired

So I ended up extending the RestClusterClient in order to use uploadJar +
runJob..you can look at the extended class at [1].
Unfortunately I still have to understand how to understand if dynamic
classloading il closed correctly or not by Job managers and Task managers
because I suspect that Tasks are not finalized correctly as detected for
Python at [2]

[1]
https://github.com/fpompermaier/flink-job-server/blob/main/flink-rest-client/src/main/java/org/apache/flink/client/program/rest/RestClusterClientExtended.java
[2] https://issues.apache.org/jira/browse/FLINK-20333

Best,
Flavio

On Wed, Dec 9, 2020 at 12:43 PM Yang Wang  wrote:

> Actually, I think the key point is that the Flink client is not friendly to
> the deployers.
> Most companies have their own deployers and I believe many of them depend
> on
> the cli commands(e.g. "flink run/run-application").
>
> I am not sure whether using the rest cluster client is the best choice. But
> we could
> have an alternative as follows.
>
> # Set the configuration based on the deployment mode(session, perjob)
> Configuration flinkConfig = new Configuration();
> ... ...
> flinkConfig.set("execution.target", "kubernetes-session");
> flinkConfig.set("kubernetes.cluster-id", "my-flink-k8s-session");
> # Build a packaged program
> PackagedProgram program =
> PackagedProgram.newBuilder().setConfiguration(flinkConfig)...build();
> # Run the packaged program on the deployment. Maybe we also need to set the
> Context.
> program.invokeInteractiveModeForExecution();
>
> In my opinion, the PackagedProgram is more appropriate for jar submission.
>
>
> Best,
> Yang
>
>
> Arvid Heise  于2020年12月8日周二 下午9:39写道：
>
> > I'm surprised that this is not possible currently. Seems like a glaring
> > missing feature to me.
> >
> > I'd assume the best way would be to extend the REST API to
> /jar/:jarId/run
> > with an option to overwrite configuration values. I'm not sure how to map
> > json well to the yaml structure of the config, but I guess we mostly have
> > simple key/value pairs anyways.
> >
> > On Tue, Dec 8, 2020 at 1:31 PM Till Rohrmann 
> wrote:
> >
> > > Hi Fabian,
> > >
> > > thanks for starting this discussion. In general I would be a bit
> hesitant
> > > to build upon Flink's web UI submission because it suffers from a
> couple
> > of
> > > drawbacks.
> > >
> > > 1) The web UI submission only supports single job applications.
> > > 2) The JobGraph is generated from within the web UI Netty thread.
> Hence,
> > if
> > > the user code blocks, then this can make the web UI unresponsive.
> > > 3) Uploaded jars are not persisted. Hence, if a JobManager failover
> > occurs
> > > between uploading and running the job, then you might have lost the
> > > uploaded jars.
> > >
> > > The reason for some of these problems is that the feature was actually
> > > implemented for some conference and almost remained untouched ever
> since.
> > > Building more functionality on top of it will mean that it will be
> harder
> > > to remove in the future.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Dec 8, 2020 at 12:00 PM Fabian Paul <
> > fabianp...@data-artisans.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Currently, the most convenient way of programmatically submitting a
> job
> > > to
> > > > a running session cluster is using Flink’s RestClusterClient.
> > > > Unfortunately, it is only supported, as of now, to submit a job
> > graph.[1]
> > > > To construct a job graph from a jar file, additional Flink
> dependencies
> > > are
> > > > required, which is not ideal.
> > > >
> > > > It is also possible to directly use the Flink rest API and first
> upload
> > > > the jar file via /jars/upload[2] and then run it via
> > /jar/:jarId/run[3].
> > > It
> > > > has the downside that it is impossible to set a Flink execution
> > > > configuration, and it will always take the underlying session cluster
> > > > configuration.
> > > >
> > > > I know changing the ClusterClient has already been discussed. It
> would
> > > > involve a lot of effort, so what do you think of making the jar
> > execution
> > > > more prominent via the rest endpoint by adding the option to pass an
> > > > execution configuration?
> > > >
> > > > Best,
> > > > Fabian
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/65cd385d7de504a946b17193aceea73b3c8e78a8/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java#L95
> > > > [2]
> > > >
> > >
> >

[jira] [Created] (FLINK-20561) Add documentation for `records-lag-max` metric.

[xiaozilong (Jira)]

xiaozilong created FLINK-20561:
--

 Summary: Add documentation for `records-lag-max` metric. 
 Key: FLINK-20561
 URL: https://issues.apache.org/jira/browse/FLINK-20561
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.12.0, 1.11.0
Reporter: xiaozilong


Currently, there are no metric description for kafka topic's lag in flink 
metrics docs. But this metric was reported in flink actually. So we should add 
some docs to guide the users to use it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20560) history-server: update archives while they're being loaded

[yazgoo (Jira)]

yazgoo created FLINK-20560:
--

 Summary: history-server: update archives while they're being loaded
 Key: FLINK-20560
 URL: https://issues.apache.org/jira/browse/FLINK-20560
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: yazgoo
 Fix For: 1.11.2
 Attachments: b.diff

When history server unpack and loads archives, we have to wait for all the 
archive to be read before updating the history server view.
If there are a lot of archives to handle, this can amount to a significant time 
with the history ui not up to date, when in fact it could show newly loaded 
jobs.

Attached is a diff of a fix for this [^b.diff]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20559) history-server: don't add caching headers for /jobs/overview.json

[yazgoo (Jira)]

yazgoo created FLINK-20559:
--

 Summary: history-server: don't add caching headers for 
/jobs/overview.json 
 Key: FLINK-20559
 URL: https://issues.apache.org/jira/browse/FLINK-20559
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.11.2
Reporter: yazgoo
 Attachments: a.diff

History server returns 
Cache-Control: max-age=300 for `/jobs/overview` path.

It should not cache this because if a new jobs get added it is not up to date.

Looking at the source code, looks like an exception was added for 
joboverview.json, but since the file was renamed, the exception was not updated.

 

Attached is a diff of a fix for handling this: [^a.diff]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-152: Hive Query Syntax Compatibility

[Rui Li]

Thanks Kurt for your inputs!

I agree we should extend Hive code to support non-Hive tables. I have
updated the wiki page to remove the limitations you mentioned, and add
typical use cases in the "Motivation" section.

Regarding comment #b, the interface is defined in flink-table-planner-blink
and only used by the blink planner. So I think "BlinkParserFactory" is a
better name, WDYT?

On Mon, Dec 7, 2020 at 12:28 PM Kurt Young  wrote:

> Thanks Rui for starting this discussion.
>
> I can see the benefit that we improve hive compatibility further, as quite
> some users are asking for this
> feature in mailing lists [1][2][3] and some online chatting tools such as
> DingTalk.
>
> I have 3 comments regarding to the design doc:
>
> a) Could you add a section to describe the typical use case you want to
> support after this feature is introduced?
> In that way, users can also have an impression how to use this feature and
> what the behavior and outcome will be.
>
> b) Regarding the naming: "BlinkParserFactory", I suggest renaming it to
> "FlinkParserFactory".
>
> c) About the two limitations you mentioned:
> 1. Only works with Hive tables and the current catalog needs to be a
> HiveCatalog.
> 2. Queries cannot involve tables/views from multiple catalogs.
> I assume this is because hive parser and analyzer doesn't support
> referring to a name with "x.y.z" fashion? Since
> we can control all the behaviors by leveraging the codes hive currently
> use. Is it possible that we can remove such
> limitations? The reason is I'm not sure if users can make the whole story
> work purely depending on hive catalog (that's
> the reason why I gave comment #a). If multiple catalogs are involved, with
> this limitation I don't think any meaningful
> pipeline could be built. For example, users want to stream data from Kafka
> to Hive, fully use hive's dialect including
> query part. The kafka table could be a temporary table or saved in default
> memory catalog.
>
>
> [1] http://apache-flink.147419.n8.nabble.com/calcite-td9059.html#a9118
> [2] http://apache-flink.147419.n8.nabble.com/hive-sql-flink-11-td9116.html
> [3]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-to-in-Flink-to-support-below-HIVE-SQL-td34162.html
>
> Best,
> Kurt
>
>
> On Wed, Dec 2, 2020 at 10:02 PM Rui Li  wrote:
>
> > Hi guys,
> >
> > I'd like to start a discussion about providing HiveQL compatibility for
> > users connecting to a hive warehouse. FLIP-123 has already covered most
> > DDLs. So now it's time to complement the other big missing part --
> queries.
> > With FLIP-152, the hive dialect covers more scenarios and makes it even
> > easier for users to migrate to Flink. More details are in the FLIP wiki
> > page [1]. Looking forward to your feedback!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-152%3A+Hive+Query+Syntax+Compatibility
> >
> > --
> > Best regards!
> > Rui Li
> >
>


-- 
Best regards!
Rui Li

[jira] [Created] (FLINK-20558) ParquetAvroStreamingFileSinkITCase.testWriteParquetAvroSpecific test failure

[Matthias (Jira)]

Matthias created FLINK-20558:


 Summary: 
ParquetAvroStreamingFileSinkITCase.testWriteParquetAvroSpecific test failure
 Key: FLINK-20558
 URL: https://issues.apache.org/jira/browse/FLINK-20558
 Project: Flink
  Issue Type: Test
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Matthias


[Build|https://dev.azure.com/mapohl/flink/_build/results?buildId=135=results]
 failed due to failing test 
\{{ParquetAvroStreamingFileSinkITCase.testWriteParquetAvroSpecific}}.
{code:java}
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.193 
s <<< FAILURE! - in 
org.apache.flink.formats.parquet.avro.ParquetAvroStreamingFileSinkITCase
[ERROR] 
testWriteParquetAvroSpecific(org.apache.flink.formats.parquet.avro.ParquetAvroStreamingFileSinkITCase)
  Time elapsed: 0.561 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.flink.formats.parquet.avro.ParquetAvroStreamingFileSinkITCase.validateResults(ParquetAvroStreamingFileSinkITCase.java:160)
at 
org.apache.flink.formats.parquet.avro.ParquetAvroStreamingFileSinkITCase.testWriteParquetAvroSpecific(ParquetAvroStreamingFileSinkITCase.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748) {code}
The assertion was caused by [this assert call checking the number of files in 
the 
bucket|https://github.com/apache/flink/blob/fdea3cdc47052d59fc20611e1be019d223d77501/flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/avro/ParquetAvroStreamingFileSinkITCase.java#L160].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20557) Support statement set in SQL CLI

[Jark Wu (Jira)]

Jark Wu created FLINK-20557:
---

 Summary: Support statement set in SQL CLI
 Key: FLINK-20557
 URL: https://issues.apache.org/jira/browse/FLINK-20557
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Client
Reporter: Jark Wu
 Fix For: 1.13.0


Support to submit multiple insert into in a single job on SQL CLI, this can be 
done by support statement set syntax in SQL CLI. 

The syntax had been discussed and reached an consensus on the mailing list: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-SQL-Syntax-for-Table-API-StatementSet-td42515.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.11.3, release candidate #1

[Yun Tang]

Hi Xintong

Thanks for driving this release! Someone reported that the checkpointed data 
size on overview page is broken [1] and I confirmed that really exists. I hope 
to include that fix in release-1.11.3 as that could be a quick fix and impact 
the usability.

BTW, I think there only existed 'release-1.11.3-rc1' tag in current code base 
instead of your descripted 'release-1.11.3-rc3' in previous mail thread.


[1] https://issues.apache.org/jira/browse/FLINK-20554

Best
Yun Tang

From: Xintong Song 
Sent: Wednesday, December 9, 2020 11:14
To: dev 
Cc: tzuli...@apache.org 
Subject: [VOTE] Release 1.11.3, release candidate #1

Hi everyone,

Please review and vote on the release candidate #1 for the version 1.11.3,
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint F8E419AA0B60C28879E876859DFF40967ABFC5A4 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.11.3-rc3" [5],
* website pull request listing the new release and adding announcement blog
post [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Xintong Song

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348761
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.3-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1405
 https://repository.apache.org/content/repositories/orgapacheflink-1406
[5] https://github.com/apache/flink/releases/tag/release-1.11.3-rc1
[6] https://github.com/apache/flink-web/pull/399