Hi Fabian,
After a bit more documentation reading I have a better understanding of how
InputFormat interface works.
Indeed I've better to wrap a custom InputFormat implementation in my source.
This article helps a lot
https://brewing.codes/2017/02/06/implementing-flink-batch-data-connector/
H François,
The TableEnvironment.connect() method can only be used if you provide
(quite a bit) more code.
It requires a TableSourceFactory and handling of all the properties that
are defined in the other builder methods. See [1].
I would recommend to either register the BatchTableSource
Hi Fabian,
I've got issues for a custom InputFormat implementation with my existing
code.
Is this can be used in combination with a BatchTableSource custom source?
As I understand your solution, I should move my source to implementations
like :
tableEnvironment
.connect(...)
Thank you Fabian,
That's good, I'll go for a custom File input stream.
All the best
François
Le lun. 4 févr. 2019 à 12:10, Fabian Hueske a écrit :
> Hi,
>
> The files will be read in a streaming fashion.
> Typically files are broken down into processing splits that are
> distributed to tasks
Hi,
The files will be read in a streaming fashion.
Typically files are broken down into processing splits that are distributed
to tasks for reading.
How a task reads a file split depends on the implementation, but usually
the format reads the split as a stream and does not read the split as a
Hi Fabian,
Thank you for this input.
This is interesting.
With such an input format, will all the file will be loaded in memory
before to be processed or will all be streamed?
All the best
François
Le mar. 29 janv. 2019 à 22:20, Fabian Hueske a écrit :
> Hi,
>
> You can point a file-based
Hi,
You can point a file-based input format to a directory and the input format
should read all files in that directory.
That works as well for TableSources that are internally use file-based
input formats.
Is that what you are looking for?
Best, Fabian
Am Mo., 28. Jan. 2019 um 17:22 Uhr
Hi all,
I'm wondering if it's possible and what's the best way to achieve the
loading of multiple files with a Json source to a JDBC sink ?
I'm running Flink 1.7.0
Let's say I have about 1500 files with the same structure (same format,
schema, everything) and I want to load them with a *batch*
