Shading in flink-table-blink and upgrade compatibility issue

2021-10-22 Thread Thomas Weise 
Hi,

As part of upgrading to Flink 1.13.3 from 1.13.2 we run into the
following problem with KafkaSource (Flink distribution is 1.13.2 and
the application was built with 1.13.3):

java.lang.NoSuchMethodError: 'void
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue,
java.util.function.Supplier, java.util.function.Consumer)'
at 
org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager.(KafkaSourceFetcherManager.java:67)
at 
org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:160)
at 
org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:127)

It turns out that flink-table-blink_2.12-1.13.2.jar contains
flink-connector-base and because that jar is under lib the 1.13.2
connector base gets picked up instead of the one bundled in the
application jar.

(The constructor in
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager
was added in 1.13.3.)

There are a few points I would like to discuss:

1) Version compatibility: A *patch* version should ideally not
introduce such a change, it should be forward and backward compatible.
Hopefully this will be the case after 1.14 with stable source API.
2) flink-table-blink - if it is meant to be self contained and usable
as a library - should not leak its shaded dependencies. It contains
FileSource and other deps from Flink, can those be relocated?
3) Do we need flink-table-blink under lib? Can it be bundled with the
application instead? It would be great if the dependencies under lib
are strictly Flink core.

Thanks,
Thomas

Re: [VOTE] FLIP-176: Unified Iteration to Support Algorithms (Flink ML)

2021-10-22 Thread Till Rohrmann 
Thanks for creating this FLIP.

+1 (binding)

Cheers,
Till


On Fri, Oct 22, 2021 at 6:29 AM Guowei Ma  wrote:

> +1 (binding)
>
> Best,
> Guowei
>
>
> On Thu, Oct 21, 2021 at 3:58 PM Yun Gao 
> wrote:
>
> >
> > Hi all,
> >
> > We would like to start the vote for FLIP-176: Unified Iteration to
> Support
> > Algorithms (Flink ML) [1].
> > This FLIP was discussed in this thread [2][3]. The FLIP-176 targets at
> > implementing the iteration
> > API in flink-ml to support the implementation of the algorithms.
> >
> > The vote will be open for at least 72 hours till 26th Oct morning,
> > including the weekend. Very thanks!
> >
> > Best,
> > Yun
> >
> > [1] https://cwiki.apache.org/confluence/x/hAEBCw
> > [2]
> >
> https://lists.apache.org/thread.html/r72e87a71b14baac3d7d16268346f5fc7c65f1de989e00b4ab2aab9ab%40%3Cdev.flink.apache.org%3E
> > [3]
> >
> https://lists.apache.org/thread.html/r63914a616de05a91dbe8e1a3208eb2b7c7c840c5c366bbd224483754%40%3Cdev.flink.apache.org%3E
>

[jira] [Created] (FLINK-24622) Unified sources do not close scheduled threads from SplitEnumeratorContext#callAsync()

2021-10-22 Thread Mason Chen (Jira) 
Mason Chen created FLINK-24622:
--

 Summary: Unified sources do not close scheduled threads from 
SplitEnumeratorContext#callAsync()
 Key: FLINK-24622
 URL: https://issues.apache.org/jira/browse/FLINK-24622
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.13.2
Reporter: Mason Chen


>From user mailing list:

I was wondering how to cancel a task that is enqueued by the callAsync() 
method, the one that takes in a time interval. For example, the KafkaSource 
uses this for topic partition discovery. It would be straightforward if the API 
returned the underlying future so that a process can cancel it. 

For Kafka, the enumerator shutdown seems to be unclean since it only closes the 
admin client and kafka consumer but not the topic partition discovery task. 
Furthermore, exceptions from that task will cause job failure and can 
potentially happen if the task is still running with the admin client closed. 
How can we address this?

This seems to be a bug with the current KafkaSource and also the unified 
Sources in general. Can you open a bug ticket in jira? I think the enumerator 
should take of first joining all the async threads before closing the 
enumerator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.13.3 released

2021-10-22 Thread Till Rohrmann 
Thanks Chesnay and Martijn for managing this release and to everyone who
contributed to it.

Cheers,
Till

On Fri, Oct 22, 2021 at 11:04 AM Yangze Guo  wrote:

> Thank Chesnay, Martijn, and everyone involved!
>
> Best,
> Yangze Guo
>
> On Fri, Oct 22, 2021 at 4:25 PM Yun Tang  wrote:
> >
> > Thanks for Chesnay & Martijn and everyone who made this release happen.
> >
> > Best
> > Yun Tang
> > 
> > From: JING ZHANG 
> > Sent: Friday, October 22, 2021 10:17
> > To: dev 
> > Cc: Martijn Visser ; Jingsong Li <
> jingsongl...@gmail.com>; Chesnay Schepler ; user <
> u...@flink.apache.org>
> > Subject: Re: [ANNOUNCE] Apache Flink 1.13.3 released
> >
> > Thank Chesnay, Martijn and every contributor for making this happen!
> >
> >
> > Thomas Weise  于2021年10月22日周五 上午12:15写道：
> >
> > Thanks for making the release happen!
> >
> > On Thu, Oct 21, 2021 at 5:54 AM Leonard Xu  wrote:
> > >
> > > Thanks to Chesnay & Martijn and everyone who made this release happen.
> > >
> > >
> > > > 在 2021年10月21日，20:08，Martijn Visser  写道：
> > > >
> > > > Thank you Chesnay, Leonard and all contributors!
> > > >
> > > > On Thu, 21 Oct 2021 at 13:40, Jingsong Li  > wrote:
> > > > Thanks, Chesnay & Martijn
> > > >
> > > > 1.13.3 really solves many problems.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Thu, Oct 21, 2021 at 6:46 PM Konstantin Knauf  > wrote:
> > > > >
> > > > > Thank you, Chesnay & Martijn, for managing this release!
> > > > >
> > > > > On Thu, Oct 21, 2021 at 10:29 AM Chesnay Schepler <
> ches...@apache.org >
> > > > > wrote:
> > > > >
> > > > > > The Apache Flink community is very happy to announce the release
> of
> > > > > > Apache Flink 1.13.3, which is the third bugfix release for the
> Apache
> > > > > > Flink 1.13 series.
> > > > > >
> > > > > > Apache Flink® is an open-source stream processing framework for
> > > > > > distributed, high-performing, always-available, and accurate data
> > > > > > streaming applications.
> > > > > >
> > > > > > The release is available for download at:
> > > > > > https://flink.apache.org/downloads.html <
> https://flink.apache.org/downloads.html>
> > > > > >
> > > > > > Please check out the release blog post for an overview of the
> > > > > > improvements for this bugfix release:
> > > > > > https://flink.apache.org/news/2021/10/19/release-1.13.3.html <
> https://flink.apache.org/news/2021/10/19/release-1.13.3.html>
> > > > > >
> > > > > > The full release notes are available in Jira:
> > > > > >
> > > > > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350329
> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350329
> >
> > > > > >
> > > > > > We would like to thank all contributors of the Apache Flink
> community
> > > > > > who made this release possible!
> > > > > >
> > > > > > Regards,
> > > > > > Chesnay
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Konstantin Knauf
> > > > >
> > > > > https://twitter.com/snntrable 
> > > > >
> > > > > https://github.com/knaufk 
> > > >
> > > >
> > > >
> > > > --
> > > > Best, Jingsong Lee
> > >
>

Re: Disable usage of TumblingEventTimeWindows when EventTime is disabled?

2021-10-22 Thread Till Rohrmann 
Hi Dario,

I think adding a sanity check that checks whether the user uses some event
time constructs and prints a warning if WatermarkStrategy.noWatermarks() is
set, could be a good improvement. It might even be ok to fail to make this
misconfiguration explicit. Hence, I think it is a good idea to open a
ticket for this problem. If you have a proposal for how to address this
issue, then post it on the ticket.

Cheers,
Till

On Thu, Oct 21, 2021 at 4:40 PM Dario Heinisch 
wrote:

> Hey there,
>
> When one uses .window(TumblingEventTimeWindows.of(SOME_TIME)) it will
> never window any values if the user uses
>
> WatermarkStrategy.>noWatermarks()
>  .withTimestampAssigner((t, timestamp) -> t.f0)
> )
>
> I was wondering whether Flink should throw an Exception at the start of
> the programming and prevent the use of it as
> no values would ever reach the process function.
>
> If so I would create a ticket and love to work on it.
>
> Here is an example:
>
> public class PlaygroundJob {
>  public static void main(String[] args) throws Exception {
>  StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new
> Configuration());
>
>  DataStreamSource> source =
> env.addSource(new SourceFunction>() {
>  @Override
>  public void run(SourceContext>
> sourceContext) throws Exception {
>  int i = 0;
>  while (true) {
>  Tuple2 tuple =
> Tuple2.of(System.currentTimeMillis(), i++ % 10);
>  sourceContext.collect(tuple);
>  }
>  }
>
>  @Override
>  public void cancel() {
>  }
>
>  });
>
>  source.assignTimestampsAndWatermarks(
>  // Switch noWatermarks() to forMonotonousTimestamps()
>  // and values are being printed.
>  WatermarkStrategy.>noWatermarks()
>  .withTimestampAssigner((t, timestamp) -> t.f0)
>  ).keyBy(t -> t.f1)
> .window(TumblingEventTimeWindows.of(Time.seconds(1)))
>  .process(new ProcessWindowFunction Integer>, String, Integer, TimeWindow>() {
>  @Override
>  public void process(Integer key, Context context,
> Iterable> iterable, Collector out) throws
> Exception {
>  int count = 0;
>  Iterator> iter =
> iterable.iterator();
>  while (iter.hasNext()) {
>  count++;
>  iter.next();
>  }
>
>  out.collect("Key: " + key + " count: " + count);
>
>  }
>  }).print();
>
>  env.execute();
>  }
> }
>
> Best regards,
>
> Dario
>
>

[jira] [Created] (FLINK-24621) JobManager fails to recover 1.13.1 checkpoint due to InflightDataRescalingDescriptor

2021-10-22 Thread Chesnay Schepler (Jira) 
Chesnay Schepler created FLINK-24621:


 Summary: JobManager fails to recover 1.13.1 checkpoint due to 
InflightDataRescalingDescriptor
 Key: FLINK-24621
 URL: https://issues.apache.org/jira/browse/FLINK-24621
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.13.3, 1.13.2
Reporter: Chesnay Schepler


A user reporter on the mailing list of a JM that is unable to read a 1.13.1 
checkpoint.

The big question is why the InflightDataRescalingDescriptor is creating 
problems, because it should not actually be contained in a checkpoint.

{code}
Caused by: org.apache.flink.util.FlinkException: Could not retrieve checkpoint 
2844 from state handle under checkpointID-0002844. This indicates 
that the retrieved state handle is broken. Try cleaning the state handle store.
at 
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore.retrieveCompletedCheckpoint(DefaultCompletedCheckpointStore.java:309)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore.recover(DefaultCompletedCheckpointStore.java:151)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedStateInternal(CheckpointCoordinator.java:1513)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreInitialCheckpointIfPresent(CheckpointCoordinator.java:1476)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:134)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:342)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:190) 
~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:122)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:132)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:110)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:340)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:317) 
~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:107)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at 
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
 ~[flink-dist_2.12-1.13.3.jar:1.13.3]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
 Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.io.InvalidClassException: 
org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor$NoRescalingDescriptor;
 local class incompatible: stream classdesc serialVersionUID = 
-5544173933105855751, local class serialVersionUID = 1
at java.io.ObjectStreamClass.initNonProxy(Unknown Source) ~[?:?]
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) ~[?:?]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24620) Add partition discovery for kafka sink when `sink.partitioner`= 'fix'

2021-10-22 Thread hehuiyuan (Jira) 
hehuiyuan created FLINK-24620:
-

 Summary: Add partition discovery for kafka sink when 
`sink.partitioner`= 'fix'
 Key: FLINK-24620
 URL: https://issues.apache.org/jira/browse/FLINK-24620
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Reporter: hehuiyuan


Add partition discovery for kafka sink when `sink.partitioner`= 'fix'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.13.3 released

2021-10-22 Thread Yangze Guo 
Thank Chesnay, Martijn, and everyone involved!

Best,
Yangze Guo

On Fri, Oct 22, 2021 at 4:25 PM Yun Tang  wrote:
>
> Thanks for Chesnay & Martijn and everyone who made this release happen.
>
> Best
> Yun Tang
> 
> From: JING ZHANG 
> Sent: Friday, October 22, 2021 10:17
> To: dev 
> Cc: Martijn Visser ; Jingsong Li 
> ; Chesnay Schepler ; user 
> 
> Subject: Re: [ANNOUNCE] Apache Flink 1.13.3 released
>
> Thank Chesnay, Martijn and every contributor for making this happen!
>
>
> Thomas Weise  于2021年10月22日周五 上午12:15写道：
>
> Thanks for making the release happen!
>
> On Thu, Oct 21, 2021 at 5:54 AM Leonard Xu  wrote:
> >
> > Thanks to Chesnay & Martijn and everyone who made this release happen.
> >
> >
> > > 在 2021年10月21日，20:08，Martijn Visser  写道：
> > >
> > > Thank you Chesnay, Leonard and all contributors!
> > >
> > > On Thu, 21 Oct 2021 at 13:40, Jingsong Li  > > > wrote:
> > > Thanks, Chesnay & Martijn
> > >
> > > 1.13.3 really solves many problems.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Oct 21, 2021 at 6:46 PM Konstantin Knauf  > > > wrote:
> > > >
> > > > Thank you, Chesnay & Martijn, for managing this release!
> > > >
> > > > On Thu, Oct 21, 2021 at 10:29 AM Chesnay Schepler  > > > >
> > > > wrote:
> > > >
> > > > > The Apache Flink community is very happy to announce the release of
> > > > > Apache Flink 1.13.3, which is the third bugfix release for the Apache
> > > > > Flink 1.13 series.
> > > > >
> > > > > Apache Flink® is an open-source stream processing framework for
> > > > > distributed, high-performing, always-available, and accurate data
> > > > > streaming applications.
> > > > >
> > > > > The release is available for download at:
> > > > > https://flink.apache.org/downloads.html 
> > > > > 
> > > > >
> > > > > Please check out the release blog post for an overview of the
> > > > > improvements for this bugfix release:
> > > > > https://flink.apache.org/news/2021/10/19/release-1.13.3.html 
> > > > > 
> > > > >
> > > > > The full release notes are available in Jira:
> > > > >
> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350329
> > > > >  
> > > > > 
> > > > >
> > > > > We would like to thank all contributors of the Apache Flink community
> > > > > who made this release possible!
> > > > >
> > > > > Regards,
> > > > > Chesnay
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Konstantin Knauf
> > > >
> > > > https://twitter.com/snntrable 
> > > >
> > > > https://github.com/knaufk 
> > >
> > >
> > >
> > > --
> > > Best, Jingsong Lee
> >

[jira] [Created] (FLINK-24619) rabbit not support " at least once "

2021-10-22 Thread szd (Jira) 
szd created FLINK-24619:
---

 Summary: rabbit not support " at least once "
 Key: FLINK-24619
 URL: https://issues.apache.org/jira/browse/FLINK-24619
 Project: Flink
  Issue Type: Bug
  Components: Connectors/ RabbitMQ
Reporter: szd


RMQSink 

 

question：message published  without confirm， so , the message may be losed

 

resolution :   add a confirmListener  or  channel.waitForConfirm ? 

 

public void invoke(IN value) {

          channel.basicPublish(.)

}

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.13.3 released

2021-10-22 Thread Yun Tang 
Thanks for Chesnay & Martijn and everyone who made this release happen.

Best
Yun Tang

From: JING ZHANG 
Sent: Friday, October 22, 2021 10:17
To: dev 
Cc: Martijn Visser ; Jingsong Li 
; Chesnay Schepler ; user 

Subject: Re: [ANNOUNCE] Apache Flink 1.13.3 released

Thank Chesnay, Martijn and every contributor for making this happen!


Thomas Weise mailto:t...@apache.org>> 于2021年10月22日周五 上午12:15写道：
Thanks for making the release happen!

On Thu, Oct 21, 2021 at 5:54 AM Leonard Xu 
mailto:xbjt...@gmail.com>> wrote:
>
> Thanks to Chesnay & Martijn and everyone who made this release happen.
>
>
> > 在 2021年10月21日，20:08，Martijn Visser 
> > mailto:mart...@ververica.com>> 写道：
> >
> > Thank you Chesnay, Leonard and all contributors!
> >
> > On Thu, 21 Oct 2021 at 13:40, Jingsong Li 
> > mailto:jingsongl...@gmail.com> 
> > >> wrote:
> > Thanks, Chesnay & Martijn
> >
> > 1.13.3 really solves many problems.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Oct 21, 2021 at 6:46 PM Konstantin Knauf 
> > mailto:kna...@apache.org> 
> > >> wrote:
> > >
> > > Thank you, Chesnay & Martijn, for managing this release!
> > >
> > > On Thu, Oct 21, 2021 at 10:29 AM Chesnay Schepler 
> > > mailto:ches...@apache.org> 
> > > >>
> > > wrote:
> > >
> > > > The Apache Flink community is very happy to announce the release of
> > > > Apache Flink 1.13.3, which is the third bugfix release for the Apache
> > > > Flink 1.13 series.
> > > >
> > > > Apache Flink® is an open-source stream processing framework for
> > > > distributed, high-performing, always-available, and accurate data
> > > > streaming applications.
> > > >
> > > > The release is available for download at:
> > > > https://flink.apache.org/downloads.html 
> > > > 
> > > >
> > > > Please check out the release blog post for an overview of the
> > > > improvements for this bugfix release:
> > > > https://flink.apache.org/news/2021/10/19/release-1.13.3.html 
> > > > 
> > > >
> > > > The full release notes are available in Jira:
> > > >
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350329
> > > >  
> > > > 
> > > >
> > > > We would like to thank all contributors of the Apache Flink community
> > > > who made this release possible!
> > > >
> > > > Regards,
> > > > Chesnay
> > > >
> > > >
> > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable 
> > >
> > > https://github.com/knaufk 
> >
> >
> >
> > --
> > Best, Jingsong Lee
>

[jira] [Created] (FLINK-24618) FlinkChillPackageRegistrar and PriorityQueueSerializer are located in wrong package

2021-10-22 Thread Matthias (Jira) 
Matthias created FLINK-24618:


 Summary: FlinkChillPackageRegistrar and PriorityQueueSerializer 
are located in wrong package
 Key: FLINK-24618
 URL: https://issues.apache.org/jira/browse/FLINK-24618
 Project: Flink
  Issue Type: Bug
  Components: API / Scala
Affects Versions: 1.14.0
Reporter: Matthias


We accidentally moved some Java classes into the wrong sub-directory as Alexey 
validly pointed out in [his PR comment for 
FLINK-23027|https://github.com/apache/flink/pull/16190#issuecomment-944283253].

`FlinkChillPackageRegistrar.java` and `PriorityQueueSerializer.java` should be 
located in `flink-scala/src/main/java` instead of `flink-scala/scr/main/scala`



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24617) Support partition keys through metadata

2021-10-22 Thread Francesco Guardiani (Jira) 
Francesco Guardiani created FLINK-24617:
---

 Summary: Support partition keys through metadata
 Key: FLINK-24617
 URL: https://issues.apache.org/jira/browse/FLINK-24617
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Runtime
Reporter: Francesco Guardiani
Assignee: Francesco Guardiani


Right now we have a lot of code around the various formats to support partition 
keys. From the FileSystemTableSource point of view, these can be handled 
reusing the metadata support provided by 
https://issues.apache.org/jira/browse/FLINK-24165. We should cleanup all that 
partition keys support code and just reuse what we have for metadata.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24616) Expose all metadata in FileStatus

2021-10-22 Thread Francesco Guardiani (Jira) 
Francesco Guardiani created FLINK-24616:
---

 Summary: Expose all metadata in FileStatus
 Key: FLINK-24616
 URL: https://issues.apache.org/jira/browse/FLINK-24616
 Project: Flink
  Issue Type: Sub-task
Reporter: Francesco Guardiani


FileStatus provides several useful metadata, including access time, creation 
time, etc. We should expose them through the support for metadata in filesystem 
connector added by https://issues.apache.org/jira/browse/FLINK-24165



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24615) Add infrastructure to support metadata in filesystem connector

2021-10-22 Thread Francesco Guardiani (Jira) 
Francesco Guardiani created FLINK-24615:
---

 Summary: Add infrastructure to support metadata in filesystem 
connector
 Key: FLINK-24615
 URL: https://issues.apache.org/jira/browse/FLINK-24615
 Project: Flink
  Issue Type: Sub-task
Reporter: Francesco Guardiani


{{FileSystemTableSource}} should implement {{SupportReadingMetadata}}, and we 
need a way to propagate file system metadata without adding the code to every 
format



--
This message was sent by Atlassian Jira
(v8.3.4#803005)