Thanks all for the discussion, I have updated FLIP-105 and FLIP-122 to use
the new format option keys.
FLIP-105:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427289
FLIP-122:
Cool, so let's go with:
format = json
json.fail-on-missing-field = true
json.ignore-parse-error = true
value.format = json
value.json.fail-on-missing-field = true
value.json.ignore-parse-error = true
Regards,
Timo
On 06.05.20 12:39, Jingsong Li wrote:
Hi,
+1 to:
format = parquet
Hi,
+1 to:
format = parquet
parquet.compression = ...
parquet.block.size = ...
parquet.page.size = ...
For the formats like parquet and orc,
Not just Flink itself, but this way also let Flink keys compatible with the
property keys of Hadoop / Hive / Spark.
And like Jark said, this way works for
Hi,
I think Timo proposed a good idea to make both side happy. That is:
format = json
json.fail-on-missing-field = true
json.ignore-parse-error = true
value.format = json
value.json.fail-on-missing-field = true
value.json.ignore-parse-error = true
This is a valid hierarchies. Besides, it's
Hi, everyone ~
Allows me to share some thoughts here.
Personally i think for SQL, "format" is obviously better than "format.name", it
is more concise and straight-forward, similar with Presto FORMAT[2] and KSQL
VALUE_FORMAT[1]; i think we move from "connector.type" to "connector" for the
same
Hi everyone,
I like Timo's proposal to organize our configuration more hierarchical
since this is what the coding guide specifies. The benefit I see is that
config options belonging to the same concept will be found in the same
nested object. Moreover, it will be possible to split the
Hi Jark,
yes, in theory every connector can design options as they like. But for
user experience and good coding style we should be consistent in Flink
connectors and configuration. Because implementers of new connectors
will copy the design of existing ones.
Furthermore, I could image that
Hi Dawid,
I just want to mention one of your response,
> What you described with
> 'format' = 'csv',
> 'csv.allow-comments' = 'true',
> 'csv.ignore-parse-errors' = 'true'
> would not work though as the `format` prefix is mandatory in the sources
as only the properties with format
> will be
Hi all,
I'd like to start with a comment that I am ok with the current state of
the FLIP-122 if there is a strong preference for it. Nevertheless I
still like the idea of adding `type` to the `format` to have it as
`format.type` = `json`.
I wanted to clarify a few things though:
@Jingsong As
Sorry for mistake,
I proposal:
connector: 'filesystem'
path: '...'
format: 'json'
format.option:
option1: '...'
option2: '...'
option3: '...'
And I think most of cases, users just need configure 'format' key, we
should make it convenient for them. There is no big problem in making
IIUC FLIP-122 already delegate the responsibility for designing and parsing
connector properties to connector developers.
So frankly speaking, no matter which style we choose, there is no strong
guarantee for either of these. So it's also possible
that developers can choose a totally different way
Here I have a little doubt. At present, our json only supports the
conventional json format. If we need to implement json with bson, json with
avro, etc., how should we express it?
Do you need like the following:
‘format.name' = 'json',
‘format.json.fail-on-missing-field' = 'false'
Thanks Timo for staring the discussion.
I am +1 for "format: 'json'".
Take a look to Dawid's yaml case:
connector: 'filesystem'
path: '...'
format: 'json'
format:
option1: '...'
option2: '...'
option3: '...'
Is this work?
According to my understanding, 'format' key is the attribute
Thanks Timo for staring the discussion.
Generally I like the idea to keep the config align with a standard like
json/yaml.
>From the user's perspective, I don't use table configs from a config file
like yaml or json for now,
And it's ok to change it to yaml like style. Actually we didn't know
Hi all,
I also wanted to share my opinion.
When talking about a ConfigOption hierarchy we use for configuring Flink
cluster I would be a strong advocate for keeping a yaml/hocon/json/...
compatible style. Those options are primarily read from a file and thus
should at least try to follow common
Personally I don't have any preference here. Compliance wih standard YAML
parser is probably more important
On Wed, Apr 29, 2020 at 5:10 PM Jark Wu wrote:
> From a user's perspective, I prefer the shorter one "format=json", because
> it's more concise and straightforward. The "kind" is
>From a user's perspective, I prefer the shorter one "format=json", because
it's more concise and straightforward. The "kind" is redundant for users.
Is there a real case requires to represent the configuration in JSON style?
As far as I can see, I don't see such requirement, and everything works
> Therefore, should we advocate instead:
>
> 'format.kind' = 'json',
> 'format.fail-on-missing-field' = 'false'
Yes. That's pretty much it.
This is reasonable important to nail down as with such violations I
believe we could not actually switch to a standard YAML parser.
On 29/04/2020 16:05,
Hi everyone,
discussions around ConfigOption seem to be very popular recently. So I
would also like to get some opinions on a different topic.
How do we represent hierarchies in ConfigOption? In FLIP-122, we agreed
on the following DDL syntax:
CREATE TABLE fs_table (
...
) WITH (
19 matches
Mail list logo
