Re: Regarding new command to download jars in flink cluster

[David Morávek]

Hi Surendra,

Since you're mentioning docker, I assume you're deploying your application
to k8s. Is that correct?

For handcrafted Kubernetes deployments, you can simply download the jar
into the user lib folder in an init container [1]. You can usually reuse
existing docker images to download the jar. For example, for S3, the AWS
CLI image will do the trick [2].

In general, my take is that this doesn't belong to the Flink itself (we
should keep the core feature matrix dense) but to the
deployment/orchestration layer around it (e.g., the AF Kubernetes Operator
[3]).

[1] https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
[2] https://hub.docker.com/r/amazon/aws-cli
[3] https://github.com/apache/flink-kubernetes-operator

Best,
D.

On Fri, Mar 3, 2023 at 8:11 PM Surendra Singh Lilhore <
surendralilh...@apache.org> wrote:

> Hi Team,
>
>
>
> According to the Flink documentation, in the APP mode, the application jar
> should be bundled with the Flink image. However, building an image for each
> new application can be difficult. Can we introduce new commands that will
> help to download the required jar locally before starting Flink JM or TM
> containers? This should be a simple command that depends on the supported
> file system (S3, HDFS, ABFS) in Flink, and the command format should be
> something like this:
>
> *./flink fs-download  *
>
> Example:
>
> *./flink fs-download
> abfs://mycontai...@storageaccount.dfs.core.windows.net/jars /tmp/appjars
>  /tmp/appjars>*
>
> I have already tested this in my cluster, and it is working fine. Before
> raising a JIRA ticket, I would like to get suggestions from the community.
>
>
> Thanks and Regards
> Surendra
>

[jira] [Created] (FLINK-31324) Broken SingleThreadFetcherManager constructor API

[Yun Tang (Jira)]

Yun Tang created FLINK-31324:


 Summary: Broken SingleThreadFetcherManager constructor API
 Key: FLINK-31324
 URL: https://issues.apache.org/jira/browse/FLINK-31324
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Parent
Reporter: Yun Tang
Assignee: Yun Tang
 Fix For: 1.17.0


FLINK-28853 changed the default constructor of {{SingleThreadFetcherManager}}. 
Though the {{SingleThreadFetcherManager}} is annotated as {{Internal}}, it 
actually acts as some-degree public API, which is widely used in many connector 
projects:
[flink-cdc-connector|https://github.com/ververica/flink-cdc-connectors/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/reader/MySqlSourceReader.java#L93],
 
[flink-connector-mongodb|https://github.com/apache/flink-connector-mongodb/blob/main/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReader.java#L58]
 and so on.

Once flink-1.17 is released, all these existing connectors are broken and 
cannot be used in new release version. Thus, I suggest to make the original 
SingleThreadFetcherManager constructor as depreacted instead of removing it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] String literal behavior in Flink

[Aitozi]

Hi,
  I encountered a problem when using string literal in Flink. Currently,
Flink will escape the string literal during codegen, so for the query below:

SELECT 'a\nb'; it will print => a\nb

then for the query

SELECT SPLIT_INDEX(col, '\n', 0);

The col can not split by the newline. If we want to split by the newline,
we should use

SELECT SPLIT_INDEX(col, '
', 0)

or

SELECT SPLIT_INDEX(col, CHR(10), 0)

The above way could be more intuitive. Some other databases support these
"Special Character Escape Sequences"[1].

In this way, we can directly use
SELECT SPLIT_INDEX(col, '\n', 0); for the query.

I know this is not standard behavior in ANSI SQL. I'm opening this thread
for some opinions from the community guys.

[1]:
https://dev.mysql.com/doc/refman/8.0/en/string-literals.html#character-escape-sequences

Thanks,
Aitozi

Re: Flink k8s native support - pod deployments and upgrades

[ramkrishna vasudevan]

Thanks Gyula for the update. This helps a lot.

Regards
Ram

On Sat, Mar 4, 2023 at 12:27 PM Gyula Fóra  wrote:

> In general there are no in place upgrades in Flink for either jobs or
> processes . This is not specific to Kubernetes, Yarn or any deployment
> mode.
>
> Gyula
>
> On Sat, 4 Mar 2023 at 06:51, ramkrishna vasudevan  >
> wrote:
>
> > Thanks for the reply. Yes I get that the operator helps out in upgrades
> > also . But when the operator works with flink k8s native clusters, it can
> > help with savepoints and helps in creating a new upgraded cluster, but
> the
> > existing cluster is not generally upgraded due to the design as how
> > clusters are created.
> >
> > Is my understanding correct?
> >
> > Regards
> > Ram
> >
> > On Fri, Mar 3, 2023 at 9:49 PM Thomas Weise  wrote:
> >
> > > Hi,
> > >
> > > The Flink k8s native integration does not handle upgrades. That's what
> > > flink-kubernates-operator was built for. Please check out:
> > >
> > >
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Fri, Mar 3, 2023 at 2:02 AM ramkrishna vasudevan <
> > > ramvasu.fl...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > The native implementation of the App mode and session mode - does not
> > > have
> > > > any replica set .
> > > > Instead it just allows the JM to create TM pods on demand.
> > > >
> > > > This is simple and easy in terms of creation of resources, but for an
> > > > upgrade story, how is this managed? Leaving K8s to manage a replica
> set
> > > > based upgrade might be easier right?
> > > >
> > > > Just wanted to understand how upgrades are handled in native K8s
> mode.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

[Ran Tao]

Hi all,

If there is no question related to this FLIP, we will start the voting
thread on 3/6 (Monday).  thank you all.

Best Regards,
Ran Tao
https://github.com/chucheng92


Ran Tao  于2023年3月1日周三 17:40写道：

> Hi, Timo. thanks.
>
> I have copied the content of docs and updated the FLIP.
>
> Thanks for your advice about consistency(ILIKE).
> Because calcite already support parse ILIKE and exist
> SqlLibraryOperators.ILIKE(which means it's not std).
> It's will not difficult to support it, and we just need to support it in
> table & sql api and not need to change sql keywords or Parser.jj.
>
> However, in flink docs, we may let users know that this is not standard,
> just for better use.
>
> Best Regards,
> Ran Tao
>
>
> Timo Walther  于2023年2月28日周二 22:23写道：
>
>> Hi Ran Tao,
>>
>> Thanks for working on this FLIP. The FLIP is in a pretty good shape
>> already and I don't have much to add.
>>
>> Will we also support ILIKE in queries? Or is this a pure DDL
>> expressions? For consistency, we should support it in SELECT and Table
>> API as well. I hope this is not too much effort. I hope everybody is
>> aware that ILIKE is not standard compliant but seems to be used by a
>> variety of vendors.
>>
>>  > Because it may be modified under discuss, we put it on the google
>> docs. please see FLIP-297: Improve Auxiliary Sql Statements Docs
>>
>> This comment confused me. It would be nice to have the Wiki page as the
>> single source of truth and abandon the Google doc. In the past we used
>> Google Docs more but nowadays I support using only the Wiki to avoid any
>> confusion.
>>
>> Regards,
>> Timo
>>
>>
>> On 28.02.23 12:51, Ran Tao wrote:
>> > thanks Sergey, sounds good.
>> > You can add in FLIP ticket[1].
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-31256
>> >
>> > Best Regards,
>> > Ran Tao
>> > https://github.com/chucheng92
>> >
>> >
>> > Sergey Nuyanzin  于2023年2月28日周二 19:44写道：
>> >
>>  Currently I think we can load from the jar and check the services
>> file to
>>  get the connector type. but is it necessary we may continue to
>> discuss.
>>  Hi, Sergey, WDYT？
>> >>
>> >> Another idea is FactoryUtil#discoverFactories and
>> >> check if it implements DynamicTableSourceFactory or
>> DynamicTableSinkFactory
>> >> with versions it could be trickier...
>> >> Moreover it seems the version could be a part of the name sometimes[1].
>> >> I think name and type could be enough or please correct me if I'm wrong
>> >>
>>  or can we open a single ticket under this FLIP?
>> >> I have a relatively old jira issue[2] for showing connectors with a
>> poc pr.
>> >> Could I propose to move this jira issue as a subtask under the FLIP
>> one and
>> >> revive it?
>> >>
>> >> [1]
>> >>
>> >>
>> https://github.com/apache/flink/blob/161014149e803bfd1d3653badb230b2ed36ce3cb/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/Factory.java#L65-L69
>> >> [2] https://issues.apache.org/jira/browse/FLINK-25788
>> >>
>> >> On Tue, Feb 28, 2023 at 11:56 AM Ran Tao 
>> wrote:
>> >>
>> >>> Hi, Jark. thanks.
>>  About ILIKE
>> >>> I have updated the FLIP for ILIKE support (Including existing
>> showTables
>> >> &
>> >>> showColumns how to change).
>> >>>
>>  About show connectors @Sergey,
>> >>> Currently I think we can load from the jar and check the services
>> file to
>> >>> get the connector type. but is it necessary we may continue to
>> discuss.
>> >>> Hi, Sergey, WDYT？or can we open a single ticket under this FLIP?
>> >>>
>> >>>
>> >>> Best Regards,
>> >>> Ran Tao
>> >>>
>> >>>
>> >>> Jark Wu  于2023年2月28日周二 17:45写道：
>> >>>
>>  Besides, if we introduce the ILIKE, we should also add this feature
>> for
>>  the previous SHOW with LIKE statements. They should be included in
>> this
>>  FLIP.
>> 
>>  Best,
>>  Jark
>> 
>> > 2023年2月28日 17:40，Jark Wu  写道：
>> >
>> > Hi Ran,
>> >
>> > Could you add descriptions about what’s the behavior and differences
>>  between the LIKE and ILIKE?
>> >
>> > Besides, I don’t see the SHOW CONNECTOR syntax and description and
>> >> how
>>  it works in the FLIP. Is it intended to be included in this FLIP?
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >> 2023年2月28日 10:58，Ran Tao  写道：
>> >>
>> >> Hi, guys. thanks for advices.
>> >>
>> >> allow me to make a small summary:
>> >>
>> >> 1.Support ILIKE
>> >> 2.Using catalog api to support show operations
>> >> 3.Need a dedicated FLIP try to support INFORMATION_SCHEMA
>> >> 4.Support SHOW CONNECTORS
>> >>
>> >> If there are no other questions, i will try to start a VOTE for
>> this
>>  FLIP.
>> >> WDYT?
>> >>
>> >> Best Regards,
>> >> Ran Tao
>> >>
>> >>
>> >> Sergey Nuyanzin  于2023年2月27日周一 21:12写道：
>> >>
>> >>> Hi Jark,
>> >>>
>> >>> thanks for your comment.
>> >>>
>>  Considering they
>>  are orthogonal and information