[jira] [Commented] (FLINK-18493) Make target home directory used to store yarn files configurable
[ https://issues.apache.org/jira/browse/FLINK-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151927#comment-17151927 ] Kevin Zhang commented on FLINK-18493: - [~fly_in_gis] Yes I'd like to work on this if one of the committers could assign this to me, thanks > Make target home directory used to store yarn files configurable > > > Key: FLINK-18493 > URL: https://issues.apache.org/jira/browse/FLINK-18493 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.10.0, 1.10.1 >Reporter: Kevin Zhang >Priority: Major > > When submitting applications to yarn, the file system's default home > directory, like /user/user_name on hdfs, is used as the base directory to > store flink files. But in different file system access control strategies, > the default home directory may be inaccessible, and it's more preferable for > users to specify thier own paths if they wish to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18493) Make target home directory used to store yarn files configurable
[ https://issues.apache.org/jira/browse/FLINK-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151860#comment-17151860 ] Kevin Zhang commented on FLINK-18493: - [~fly_in_gis] yes I mean the staging directory, so what's blocking the issue? It seems the issue has been idle for months > Make target home directory used to store yarn files configurable > > > Key: FLINK-18493 > URL: https://issues.apache.org/jira/browse/FLINK-18493 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.10.0, 1.10.1 >Reporter: Kevin Zhang >Priority: Major > > When submitting applications to yarn, the file system's default home > directory, like /user/user_name on hdfs, is used as the base directory to > store flink files. But in different file system access control strategies, > the default home directory may be inaccessible, and it's more preferable for > users to specify thier own paths if they wish to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18493) Make target home directory used to store yarn files configurable
[ https://issues.apache.org/jira/browse/FLINK-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated FLINK-18493: Description: When submitting applications to yarn, the file system's default home directory, like /user/user_name on hdfs, is used as the base directory to store flink files. But in different file system access control strategies, the default home directory may be inaccessible, and it's more preferable for users to specify thier own paths if they wish to. (was: When submitting applications to yarn, the file system's default home directory, like /user/user_name on hdfs, is used as the base directory to store flink files. But in different file system access control strategies, the default home directory may be not inaccessible, and it's more preferable for users to specify thier own paths if they wish to.) > Make target home directory used to store yarn files configurable > > > Key: FLINK-18493 > URL: https://issues.apache.org/jira/browse/FLINK-18493 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.10.0, 1.10.1 >Reporter: Kevin Zhang >Priority: Major > > When submitting applications to yarn, the file system's default home > directory, like /user/user_name on hdfs, is used as the base directory to > store flink files. But in different file system access control strategies, > the default home directory may be inaccessible, and it's more preferable for > users to specify thier own paths if they wish to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18493) Make target home directory used to store yarn files configurable
Kevin Zhang created FLINK-18493: --- Summary: Make target home directory used to store yarn files configurable Key: FLINK-18493 URL: https://issues.apache.org/jira/browse/FLINK-18493 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.10.1, 1.10.0 Reporter: Kevin Zhang When submitting applications to yarn, the file system's default home directory, like /user/user_name on hdfs, is used as the base directory to store flink files. But in different file system access control strategies, the default home directory may be not inaccessible, and it's more preferable for users to specify thier own paths if they wish to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18211) Dynamic properties setting 'pipeline.jars' will be overwritten
[ https://issues.apache.org/jira/browse/FLINK-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136434#comment-17136434 ] Kevin Zhang commented on FLINK-18211: - hi [~kkl0u] IIRC, there is no other good way to ship jars, so maybe just using `pipeline.jars` for shipping jars is a good option, what do you think about it? > Dynamic properties setting 'pipeline.jars' will be overwritten > -- > > Key: FLINK-18211 > URL: https://issues.apache.org/jira/browse/FLINK-18211 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.10.0, 1.11.0 >Reporter: Echo Lee >Assignee: Echo Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > When we submit the application through "flink run > -Dpipeline.jars='/user1.jar, user2.jar'..." command, configuration will > include 'pipeline.jars', But ExecutionConfigAccessor#fromProgramOptions will > be reset this property, So the property set by the user is invalid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110793#comment-17110793 ] Kevin Zhang commented on FLINK-17745: - [~fly_in_gis] IIUC, the problem that [~Echo Lee] addresses is that both the fat jar and the jars extracted from the fat jar are added to `pipeline.jars`. Obviously this is retundant, and also I agree with lisen that only jars extraced from the fat jar should be added to `pipeline.jars`. [~kkl0u]'s pr seems only set the fat jar to `pipeline.jars`, I'm not sure whether it works > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Affects Versions: 1.11.0 >Reporter: lisen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14567) Aggregate query with more than two group fields can't be write into HBase sink
[
https://issues.apache.org/jira/browse/FLINK-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974992#comment-16974992
]
Kevin Zhang commented on FLINK-14567:
-
I basically agree with [~jackylau]. In the proposed scenario flink may be not
able to figure out whether concating two unique key fields with a specific
separator can preserve the uniqueness. However, most of the time the users
understand their data and know using which separtor won't break the uniqueness,
at this time we should let the users to determine whether the query result can
be inserted into hbase sink. The solution may be providing another
concat_ws-like function with an additional parameter to indicate if to ignore
the potential risks, and document the reason properly.
> Aggregate query with more than two group fields can't be write into HBase sink
> --
>
> Key: FLINK-14567
> URL: https://issues.apache.org/jira/browse/FLINK-14567
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Legacy Planner, Table
> SQL / Planner
>Reporter: Jark Wu
>Priority: Critical
> Fix For: 1.10.0
>
>
> If we have a hbase table sink with rowkey of varchar (also primary key) and a
> column of bigint, we want to write the result of the following query into the
> sink using upsert mode. However, it will fail when primary key check with the
> exception "UpsertStreamTableSink requires that Table has a full primary keys
> if it is updated."
> {code:sql}
> select concat(f0, '-', f1) as key, sum(f2)
> from T1
> group by f0, f1
> {code}
> This happens in both blink planner and old planner. That is because if the
> query works in update mode, then there must be a primary key exist to be
> extracted and set to {{UpsertStreamTableSink#setKeyFields}}.
> That's why we want to derive primary key for concat in FLINK-14539, however,
> we found that the primary key is not preserved after concating. For example,
> if we have a primary key (f0, f1, f2) which are all varchar type, say we have
> two unique records ('a', 'b', 'c') and ('ab', '', 'c'), but the results of
> concat(f0, f1, f2) are the same, which means the concat result is not primary
> key anymore.
> So here comes the problem, how can we proper support HBase sink or such use
> case?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Zhang closed FLINK-14539.
---
Resolution: Invalid
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Assignee: Kevin Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962759#comment-16962759
]
Kevin Zhang commented on FLINK-14539:
-
[~jark] That's good, I'd like to join the discussion by then, thanks.
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Assignee: Kevin Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962753#comment-16962753
]
Kevin Zhang commented on FLINK-14539:
-
[~jark] thansk, greate catch, I omitted this kind of situations. I'll close the
PR later.
But there are some scenarios where we need preserve the unique keys. For
example, we have a hbase table sink with rowkey of varchar (also primary key)
and a column of bigint, we want to write the result of the following query into
the sink using upsert mode, currently the sql will fail the primary key check,
do you have any suggestions about how to do this?
{code:sql}
select f0, f1 sum(f2)
from t1
group by f0, f1
{code}
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Assignee: Kevin Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960875#comment-16960875
]
Kevin Zhang edited comment on FLINK-14539 at 10/28/19 9:06 AM:
---
Thanks for your opinions [~jark] [~danny0405]. Actually I've already implement
this by using RexBiVisitor just like a RexVisitor but using the additional
argument to pass the outIndex, otherwise it's hard to determine what inIndex
and outIndex pair should be put into the mapInToOutPos. I'll open a pr and
appreciate it if you can help to review there.
was (Author: kevinzwx):
Thanks for your opinions[~jark][~danny0405]. Actually I've already implement
this by using RexBiVisitor just like a RexVisitor but using the additional
argument to pass the outIndex, otherwise it's hard to determine what inIndex
and outIndex pair should be put into the mapInToOutPos. I'll open a pr and
appreciate it if you can help to review there.
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Priority: Major
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960875#comment-16960875
]
Kevin Zhang commented on FLINK-14539:
-
Thanks for your opinions[~jark][~danny0405]. Actually I've already implement
this by using RexBiVisitor just like a RexVisitor but using the additional
argument to pass the outIndex, otherwise it's hard to determine what inIndex
and outIndex pair should be put into the mapInToOutPos. I'll open a pr and
appreciate it if you can help to review there.
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Priority: Major
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Zhang updated FLINK-14539:
Description:
Currently unique key metadata of a project relnode are only kept in the
following three situations:
# project the child unique keys while not changing them
# cast the child unique key when ignoring nulls and the original type of the
field and cast type are the same
# rename the child unique keys
Besides these situations, concat and concat_ws should also keep the metadata if
they won't break the uniqueness of the child unique keys, i.e. each operands is
in one of the above situations, and the operands include all the child unique
keys.
Say the f0 and f1 are the unique key fields of the child node, the following
sqls should keep the unique key metadata
{code:sql}
select concat(f0, f1)
-- the type of f0 and f1 are both varchar originally and ignore nulls
select concat(cast(f0 as varchar), f1)
select cast(concat(f0, f1) as varchar)
{code}
while the following sqls should discard the unique key metadata
{code:sql}
-- the type of f0 and f1 are both varchar originally
select concat(cast(f0 as bigint), f1)
select cast(concat(f0, f1) as bigint)
{code}
was:
Currently unique key metadata of a project relnode are only kept in the
following three situations:
# project the child unique keys while not changing them
# cast the child unique key when ignoring nulls and the original type of the
field and cast type are the same
# rename the child unique keys
Besides these situations, concat and concat_ws should also keep the metadata if
they won't break the uniqueness of the child unique keys, i.e. each operands is
in one of the above situations, and the operands include all the child unique
keys.
Say the f0 and f1the child are the unique keys of the child node, the following
sqls should keep the unique key metadata
{code:sql}
select concat(f0, f1)
-- the type of f0 and f1 are both varchar originally and ignore nulls
select concat(cast(f0 as varchar), f1)
select cast(concat(f0, f1) as varchar)
{code}
while the following sqls should discard the unique key metadata
{code:sql}
-- the type of f0 and f1 are both varchar originally
select concat(cast(f0 as bigint), f1)
select cast(concat(f0, f1) as bigint)
{code}
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Priority: Major
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1 are the unique key fields of the child node, the following
> sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960781#comment-16960781
]
Kevin Zhang edited comment on FLINK-14539 at 10/28/19 6:12 AM:
---
I indend to implement this using a RexBiVisitor, because there are cases where
concat and cast can call each other inside, and it's more convenient to extend
when we find more cases that we can keep the unique key metadata. If it's ok
and not breaks some other issues, I'd like to open a pr for further review.
was (Author: kevinzwx):
I indend to implement this using a RexBiVisitor, because there are cases concat
and cast can each other inside, and it's more convenient to extend when we find
more cases that we can keep the unique key metadata. If it's ok and not breaks
some other issues, I'd like to open a pr for further review.
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Priority: Major
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1the child are the unique keys of the child node, the
> following sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
[
https://issues.apache.org/jira/browse/FLINK-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960781#comment-16960781
]
Kevin Zhang commented on FLINK-14539:
-
I indend to implement this using a RexBiVisitor, because there are cases concat
and cast can each other inside, and it's more convenient to extend when we find
more cases that we can keep the unique key metadata. If it's ok and not breaks
some other issues, I'd like to open a pr for further review.
> Unique key metadata should be ketp when using concat or concat_ws in some
> cases
> ---
>
> Key: FLINK-14539
> URL: https://issues.apache.org/jira/browse/FLINK-14539
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Kevin Zhang
>Priority: Major
>
> Currently unique key metadata of a project relnode are only kept in the
> following three situations:
> # project the child unique keys while not changing them
> # cast the child unique key when ignoring nulls and the original type of the
> field and cast type are the same
> # rename the child unique keys
> Besides these situations, concat and concat_ws should also keep the metadata
> if they won't break the uniqueness of the child unique keys, i.e. each
> operands is in one of the above situations, and the operands include all the
> child unique keys.
> Say the f0 and f1the child are the unique keys of the child node, the
> following sqls should keep the unique key metadata
> {code:sql}
> select concat(f0, f1)
> -- the type of f0 and f1 are both varchar originally and ignore nulls
> select concat(cast(f0 as varchar), f1)
> select cast(concat(f0, f1) as varchar)
> {code}
> while the following sqls should discard the unique key metadata
> {code:sql}
> -- the type of f0 and f1 are both varchar originally
> select concat(cast(f0 as bigint), f1)
> select cast(concat(f0, f1) as bigint)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14539) Unique key metadata should be ketp when using concat or concat_ws in some cases
Kevin Zhang created FLINK-14539:
---
Summary: Unique key metadata should be ketp when using concat or
concat_ws in some cases
Key: FLINK-14539
URL: https://issues.apache.org/jira/browse/FLINK-14539
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.9.1, 1.9.0
Reporter: Kevin Zhang
Currently unique key metadata of a project relnode are only kept in the
following three situations:
# project the child unique keys while not changing them
# cast the child unique key when ignoring nulls and the original type of the
field and cast type are the same
# rename the child unique keys
Besides these situations, concat and concat_ws should also keep the metadata if
they won't break the uniqueness of the child unique keys, i.e. each operands is
in one of the above situations, and the operands include all the child unique
keys.
Say the f0 and f1the child are the unique keys of the child node, the following
sqls should keep the unique key metadata
{code:sql}
select concat(f0, f1)
-- the type of f0 and f1 are both varchar originally and ignore nulls
select concat(cast(f0 as varchar), f1)
select cast(concat(f0, f1) as varchar)
{code}
while the following sqls should discard the unique key metadata
{code:sql}
-- the type of f0 and f1 are both varchar originally
select concat(cast(f0 as bigint), f1)
select cast(concat(f0, f1) as bigint)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang closed FLINK-14436. --- Resolution: Won't Fix > Add getter for ContextEnvironment instance in StreamContextEnvironment > -- > > Key: FLINK-14436 > URL: https://issues.apache.org/jira/browse/FLINK-14436 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: Kevin Zhang >Priority: Major > Labels: easyfix > > Currently in StreamContextEnvironment the member variable ctx, an instance of > ContextEnvironment, is private. It's very helpful to add a getter for ctx > because through the ContextEnvironment we can access the ClusterClient object > of the application, and then get the information of the flink jobs or > submit/cancel jobs. It's especially useful for developers building a server > upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954317#comment-16954317 ] Kevin Zhang commented on FLINK-14436: - Thanks [~tison] for the information, I'll close this issue. > Add getter for ContextEnvironment instance in StreamContextEnvironment > -- > > Key: FLINK-14436 > URL: https://issues.apache.org/jira/browse/FLINK-14436 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: Kevin Zhang >Priority: Major > Labels: easyfix > > Currently in StreamContextEnvironment the member variable ctx, an instance of > ContextEnvironment, is private. It's very helpful to add a getter for ctx > because through the ContextEnvironment we can access the ClusterClient object > of the application, and then get the information of the flink jobs or > submit/cancel jobs. It's especially useful for developers building a server > upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14436) Add getter for ContextEnvironment instance in StreamContextEnvironment
Kevin Zhang created FLINK-14436: --- Summary: Add getter for ContextEnvironment instance in StreamContextEnvironment Key: FLINK-14436 URL: https://issues.apache.org/jira/browse/FLINK-14436 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.9.0 Reporter: Kevin Zhang Currently in StreamContextEnvironment the member variable ctx, an instance of ContextEnvironment, is private. It's very helpful to add a getter for ctx because through the ContextEnvironment we can access the ClusterClient object of the application, and then get the information of the flink jobs or submit/cancel jobs. It's especially useful for developers building a server upon flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13547) Verify and correct string function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916313#comment-16916313 ] Kevin Zhang commented on FLINK-13547: - Thanks [~docete] and [~jark], we will have a look at the related issues. > Verify and correct string function's semantic for Blink planner > --- > > Key: FLINK-13547 > URL: https://issues.apache.org/jira/browse/FLINK-13547 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently many string builtin functions in blink planner follow hive/spark > semantics, which should keep compatible with old planner. And some > non-standard functions(Blink planner intros) should be removed. > * concat/concat_ws function (null treatment) > * substring function (follow calcite/flink) > * from_base64 should return string not binary > * intro truncate function to blink planner > * uuid should be no-argument (remove the one-argument version) > * length/jsonvalue/keyvalue/substr (non-standard function should be removed) > * md5/sha1/sha2/sha224/sha256/sha384/sha512(remove the two-arguments version) > * ascii (operand type should beSqlTypeFamily.CHARACTER) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (FLINK-13547) Verify and correct string function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914052#comment-16914052 ] Kevin Zhang edited comment on FLINK-13547 at 8/23/19 8:33 AM: -- Hi [~docete], [~jark]，is there substitute standard functions for the removed functions, like jsonvalue? Some of these removed functions are very useful and we are using them in our project, otherwise we could only add these functions by ourselves. was (Author: kevinzwx): Hi [~docete] [~jark]，is there substitute standard functions for the removed functions, like jsonvalue? Some of these removed functions are very useful and we are using them in our project, otherwise we could only add these functions by ourselves. > Verify and correct string function's semantic for Blink planner > --- > > Key: FLINK-13547 > URL: https://issues.apache.org/jira/browse/FLINK-13547 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently many string builtin functions in blink planner follow hive/spark > semantics, which should keep compatible with old planner. And some > non-standard functions(Blink planner intros) should be removed. > * concat/concat_ws function (null treatment) > * substring function (follow calcite/flink) > * from_base64 should return string not binary > * intro truncate function to blink planner > * uuid should be no-argument (remove the one-argument version) > * length/jsonvalue/keyvalue/substr (non-standard function should be removed) > * md5/sha1/sha2/sha224/sha256/sha384/sha512(remove the two-arguments version) > * ascii (operand type should beSqlTypeFamily.CHARACTER) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13547) Verify and correct string function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914052#comment-16914052 ] Kevin Zhang commented on FLINK-13547: - Hi [~docete] [~jark]，is there substitute standard functions for the removed functions, like jsonvalue? Some of these removed functions are very useful and we are using them in our project, otherwise we could only add these functions by ourselves. > Verify and correct string function's semantic for Blink planner > --- > > Key: FLINK-13547 > URL: https://issues.apache.org/jira/browse/FLINK-13547 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently many string builtin functions in blink planner follow hive/spark > semantics, which should keep compatible with old planner. And some > non-standard functions(Blink planner intros) should be removed. > * concat/concat_ws function (null treatment) > * substring function (follow calcite/flink) > * from_base64 should return string not binary > * intro truncate function to blink planner > * uuid should be no-argument (remove the one-argument version) > * length/jsonvalue/keyvalue/substr (non-standard function should be removed) > * md5/sha1/sha2/sha224/sha256/sha384/sha512(remove the two-arguments version) > * ascii (operand type should beSqlTypeFamily.CHARACTER) -- This message was sent by Atlassian Jira (v8.3.2#803003)
