[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime. # We need to define the resolution order # Alternatively, we can try to match Hive built-in functions in Flink and eliminate the need of supporting them. The gap needs to be discovered first. cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime. cc [~xuefuz] [~lirui] [~Terry1897] > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime. > # We need to define the resolution order > # Alternatively, we can try to match Hive built-in functions in Flink and > eliminate the need of supporting them. The gap needs to be discovered first. > > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime. # We need to define the resolution order # Alternatively, we can try to match Hive built-in functions in Flink and eliminate the need of supporting them. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files. According my sampling in the 195 functions, some are straight-forward to rewrite, some don't seem to be frequently used. cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime. # We need to define the resolution order # Alternatively, we can try to match Hive built-in functions in Flink and eliminate the need of supporting them. The gap needs to be discovered first. cc [~xuefuz] [~lirui] [~Terry1897] > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime. > # We need to define the resolution order > # Alternatively, we can try to match Hive built-in functions in Flink and > eliminate the need of supporting them. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > Please see attached files. According my sampling in the 195 functions, some > are straight-forward to rewrite, some don't seem to be frequently used. > > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime. cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. First and basic option is to do it the hard way by integrating hive's function registry, which architecturely can be hard. Second option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files. According my sampling in the 195 functions, some are straight-forward to rewrite, some don't seem to be frequently used. Besides rewriting all of them, another option for users is to manually register those builtin functions in Hive metastore, so Flink can load them thru HiveCatalog at runtime. Lastly, we can load and hold hive builtin functions in an in-memory map of HiveCatalog as if it's from hive function registry cc [~xuefuz] [~lirui] [~Terry1897] > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime. > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. First and basic option is to do it the hard way by integrating hive's function registry, which architecturely can be hard. Second option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files. According my sampling in the 195 functions, some are straight-forward to rewrite, some don't seem to be frequently used. Besides rewriting all of them, another option for users is to manually register those builtin functions in Hive metastore, so Flink can load them thru HiveCatalog at runtime. Lastly, we can load and hold hive builtin functions in an in-memory map of HiveCatalog as if it's from hive function registry cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files. According my sampling in the 195 functions, some are straight-forward to rewrite, some don't seem to be frequently used. Besides rewriting all of them, another option for users is to manually register those builtin functions in Hive metastore, so Flink can load them thru HiveCatalog at runtime. cc [~xuefuz] [~lirui] [~Terry1897] > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > First and basic option is to do it the hard way by integrating hive's > function registry, which architecturely can be hard. > Second option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > Please see attached files. According my sampling in the 195 functions, some > are straight-forward to rewrite, some don't seem to be frequently used. > Besides rewriting all of them, another option for users is to manually > register those builtin functions in Hive metastore, so Flink can load them > thru HiveCatalog at runtime. > Lastly, we can load and hold hive builtin functions in an in-memory map of > HiveCatalog as if it's from hive function registry > > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files. According my sampling in the 195 functions, some are straight-forward to rewrite, some don't seem to be frequently used. Besides rewriting all of them, another option for users is to manually register those builtin functions in Hive metastore, so Flink can load them thru HiveCatalog at runtime. cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files cc [~xuefuz] [~lirui] [~Terry1897] > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > One option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > Please see attached files. According my sampling in the 195 functions, some > are straight-forward to rewrite, some don't seem to be frequently used. > Besides rewriting all of them, another option for users is to manually > register those builtin functions in Hive metastore, so Flink can load them > thru HiveCatalog at runtime. > > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. Please see attached files cc [~xuefuz] [~lirui] [~Terry1897] was: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > One option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > Please see attached files > > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Attachment: hive builtin functions that are missing in flink.txt > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > One option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Attachment: (was: hive builtin functions that are missing in flink.txt) > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > One option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Description: Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are registered into in-memory function catalog at runtime, which makes it hard for Flink to integrate with architecturely. One option to support rich Hive built-in functions is to develop builtin functions in Flink with the same logic. I did a simple comparison. With Flink 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > > Unlike UDFs that are persisted in Hive Metastore, Hive builtin functions are > registered into in-memory function catalog at runtime, which makes it hard > for Flink to integrate with architecturely. > One option to support rich Hive built-in functions is to develop builtin > functions in Flink with the same logic. I did a simple comparison. With Flink > 1.10.0 and Hive 2.3.4, they have 56 common (of same name) built-in functions; > there are 195 functions in Hive 2.3.4 that don't exist in Flink 1.10.0. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (FLINK-13755) support Hive built-in functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-13755: - Attachment: common builtin functions is flink and hive.txt hive builtin functions that are missing in flink.txt > support Hive built-in functions in Flink > > > Key: FLINK-13755 > URL: https://issues.apache.org/jira/browse/FLINK-13755 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.10.0 > > Attachments: common builtin functions is flink and hive.txt, hive > builtin functions that are missing in flink.txt > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
