Thanks everyone for the positive feedback.
I updated the FLIP with the proposed minimal solution:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Make+expanding+behavior+of+virtual+metadata+columns+configurable
I also changed the title to not cause any confusion with the concept
SGTM
For future reference, responding here:
5) Any operator can introduce system (psedo) columns.
>
> This is clearly out of scope for this FLIP. The implementation effort
> would be huge and could introduce a lot of bugs.
>
I didn't imply any specific implementation or feature coverage in the
Hi Timo,
Your last suggestion sounds good.
Best regards,
Jing
On Mon, Aug 21, 2023 at 4:21 AM Benchao Li wrote:
> It sounds good to me too, that we avoid introducing the concept of "system
> columns" for now.
>
> Timo Walther 于2023年8月18日周五 22:38写道：
>
> > Great, I also like my last suggestion
It sounds good to me too, that we avoid introducing the concept of "system
columns" for now.
Timo Walther 于2023年8月18日周五 22:38写道：
> Great, I also like my last suggestion as it is even more elegant. I will
> update the FLIP until Monday.
>
> Regards,
> Timo
>
> On 17.08.23 13:55, Jark Wu wrote:
>
Great, I also like my last suggestion as it is even more elegant. I will
update the FLIP until Monday.
Regards,
Timo
On 17.08.23 13:55, Jark Wu wrote:
Hi Timo,
I'm fine with your latest suggestion that introducing a flag to control
expanding behavior of metadata virtual columns, but not
Hi Timo,
I'm fine with your latest suggestion that introducing a flag to control
expanding behavior of metadata virtual columns, but not introducing
any concept of system/pseudo columns for now.
Best,
Jark
On Tue, 15 Aug 2023 at 23:25, Timo Walther wrote:
> Hi everyone,
>
> I would like to
Hi everyone,
I would like to bump this thread up again.
Esp. I would like to hear opinions on my latest suggestions to simply
use METADATA VIRTUAL as system columns and only introduce a config
option for the SELECT * behavior. Implementation-wise this means minimal
effort and less new
Hi everyone,
thanks for the various feedback and lively discussion. Sorry, for the
late reply as I was on vacation. Let me answer to some of the topics:
1) Systems columns should not be shown with DESCRIBE statements
This sounds fine to me. I will update the FLIP in the next iteration.
2)
Looks like both kinds of system columns can converge.
We can say that any operator can introduce system (psedo) columns.
cc Eugene who is also interested in the subject.
On Wed, Aug 2, 2023 at 1:03 AM Paul Lam wrote:
> Hi Timo,
>
> Thanks for starting the discussion! System columns are no
Hi Timo,
Thanks for starting the discussion! System columns are no doubt a
good boost on Flink SQL’s usability, and I see the feedbacks are
mainly concerns about the accessibility of system columns.
I think most of the concerns could be solved by clarifying the
ownership of the system columns.
Hi Timo,
Thanks for your proposal. I think this is a nice feature for users and I
prefer option 3.
I only have one concern about the concept of pseudo-column or
system-column,
because this is the first time we introduce it in Flink SQL. The
confusion is similar to the
question of Benchao and
>
> `SELECT * FROM (SELECT $rowtime, * FROM t);`
> Am I right that it will show `$rowtime` in output ?
Yes, all explicitly selected columns become a part of the result (and
intermediate) schema, and hence propagate.
On Thu, Jul 27, 2023 at 2:40 PM Alexey Leonov-Vendrovskiy <
Thank you, Timo, for starting this FLIP!
I propose the following change:
Remove the requirement that DESCRIBE need to show system columns.
Some concrete vendor specific catalog implementations might prefer this
approach.
Usually the same system columns are available on all (or family) of
Hi Timo,
Thanks for the FLIP.
I also tend to think that Option 3 is better.
I would be also interested in a question mentioned by Benchao Li.
And a similar question about nested queries like
`SELECT * FROM (SELECT $rowtime, * FROM t);`
Am I right that it will show `$rowtime` in output ?
On
Hi Timo,
Thanks for the FLIP, I also like the idea and option 3 sounds good to me.
I would like to discuss a case which is not mentioned in the current FLIP.
How are the "System column"s expressed in intermediate result, e.g. Join?
E.g. `SELECT * FROM t1 JOIN t2`, I guess it should not include
Hi Timo,
Thanks for your proposal. It is a very pragmatic feature. Among all options
in the FLIP, option 3 is one I prefer too and I'd like to ask some
questions to understand your thoughts.
1. I did some research on pseudo columns, just out of curiosity, do you
know why most SQL systems do not
Hi Timo,
this makes sense to me. Option 3 seems reasonable, too.
Cheers,
Konstantin
Am Di., 25. Juli 2023 um 12:53 Uhr schrieb Timo Walther :
> Hi everyone,
>
> I would like to start a discussion about introducing the concept of
> "System Columns" in SQL and Table API.
>
> The subject sounds
Hi everyone,
I would like to start a discussion about introducing the concept of
"System Columns" in SQL and Table API.
The subject sounds bigger than it actually is. Luckily, Flink SQL
already exposes the concept of metadata columns. And this proposal is
just a slight adjustment for how
18 matches
Mail list logo
