[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569099#comment-17569099
]
Jing Zhang commented on FLINK-24862:
[~hehuiyuan] Thanks for patch. Merged.
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0
>Reporter: xiangqiao
>Assignee: xiangqiao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.6
>
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> When hive udaf/udtf is used, a validate exception is thrown ，i added a unit
> test in HiveDialectITCase to reproduce this question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png|width=1363,height=282!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569098#comment-17569098
]
Jing Zhang commented on FLINK-24862:
Fixed in release-1.14: b7e3eb948017e50b3d8f9f7ef3f94330bea71400
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0
>Reporter: xiangqiao
>Assignee: xiangqiao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.6
>
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> When hive udaf/udtf is used, a validate exception is thrown ，i added a unit
> test in HiveDialectITCase to reproduce this question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png|width=1363,height=282!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17507306#comment-17507306
]
hehuiyuan commented on FLINK-24862:
---
[~jingzhang] [~jark] , are there any plans to merge 1.14 ?
https://github.com/apache/flink/pull/19069
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0
>Reporter: xiangqiao
>Assignee: xiangqiao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> When hive udaf/udtf is used, a validate exception is thrown ，i added a unit
> test in HiveDialectITCase to reproduce this question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png|width=1363,height=282!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460378#comment-17460378
]
Jing Zhang commented on FLINK-24862:
Fixed in master: 95b8309213a5470b2e7e61771d6e90de75c71227
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0, 1.14.0
>Reporter: xiangqiao
>Assignee: xiangqiao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> When hive udaf/udtf is used, a validate exception is thrown ，i added a unit
> test in HiveDialectITCase to reproduce this question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png|width=1363,height=282!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442054#comment-17442054
]
xiangqiao commented on FLINK-24862:
---
Thank you [~jark] ,remove the "temporary" keyword is to create a global
function, which will be written to hive Metastore, which can not meet our needs.
I have solved this problem and can work normally now. Can you review it for me?
I'm not sure if it will cause other problems. THX.
https://github.com/apache/flink/pull/17761
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0, 1.14.0
>Reporter: xiangqiao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> Here are two questions：
> 1.First question, I added a unit test in HiveDialectITCase to reproduce this
> question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png!
> 2.When I solved the first problem, I met the second problem，I added a unit
> test in HiveDialectITCase to reproduce this question:
> This is the compatibility of hive udtf. Refer to this
> issue：https://issues.apache.org/jira/browse/HIVE-5737
> {code:java}
> @Test
> public void testTemporaryFunctionUDTFInitializeWithStructObjectInspector()
> throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_split as '%s'",
>
> HiveGenericUDTFTest.TestSplitUDTFInitializeWithStructObjectInspector.class
> .getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_split"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x string)");
> tableEnv.executeSql("insert into src values ('a,b,c')").await();
> assertEquals(
> "[+I[a], +I[b], +I[c]]",
> queryResult(tableEnv.sqlQuery("select temp_split(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[a], +I[b], +I[c]]",
> queryResult(tableEnv.sqlQuery("select temp_split(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_split");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-21-04-32-660.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24862) The user-defined hive udaf/udtf cannot be used normally in hive dialect
[
https://issues.apache.org/jira/browse/FLINK-24862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441744#comment-17441744
]
Jark Wu commented on FLINK-24862:
-
AFAIK, create temporary function for hive functions is not supported. You can
try to remove the "temporary" keyword which should work.
> The user-defined hive udaf/udtf cannot be used normally in hive dialect
> ---
>
> Key: FLINK-24862
> URL: https://issues.apache.org/jira/browse/FLINK-24862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0, 1.14.0
>Reporter: xiangqiao
>Priority: Major
> Attachments: image-2021-11-10-20-55-11-988.png,
> image-2021-11-10-21-04-32-660.png
>
>
> Here are two questions：
> 1.First question, I added a unit test in HiveDialectITCase to reproduce this
> question:
> {code:java}
> @Test
> public void testTemporaryFunctionUDAF() throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_count as '%s'",
> GenericUDAFCount.class.getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_count"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x int)");
> tableEnv.executeSql("insert into src values (1),(-1)").await();
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[2]]",
> queryResult(tableEnv.sqlQuery("select temp_count(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_count");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-20-55-11-988.png!
> 2.When I solved the first problem, I met the second problem，I added a unit
> test in HiveDialectITCase to reproduce this question:
> This is the compatibility of hive udtf. Refer to this
> issue：https://issues.apache.org/jira/browse/HIVE-5737
> {code:java}
> @Test
> public void testTemporaryFunctionUDTFInitializeWithStructObjectInspector()
> throws Exception {
> // create temp function
> tableEnv.executeSql(
> String.format(
> "create temporary function temp_split as '%s'",
>
> HiveGenericUDTFTest.TestSplitUDTFInitializeWithStructObjectInspector.class
> .getName()));
> String[] functions = tableEnv.listUserDefinedFunctions();
> assertArrayEquals(new String[] {"temp_split"}, functions);
> // call the function
> tableEnv.executeSql("create table src(x string)");
> tableEnv.executeSql("insert into src values ('a,b,c')").await();
> assertEquals(
> "[+I[a], +I[b], +I[c]]",
> queryResult(tableEnv.sqlQuery("select temp_split(x) from
> src")).toString());
> // switch DB and the temp function can still be used
> tableEnv.executeSql("create database db1");
> tableEnv.useDatabase("db1");
> assertEquals(
> "[+I[a], +I[b], +I[c]]",
> queryResult(tableEnv.sqlQuery("select temp_split(x) from
> `default`.src"))
> .toString());
> // drop the function
> tableEnv.executeSql("drop temporary function temp_split");
> functions = tableEnv.listUserDefinedFunctions();
> assertEquals(0, functions.length);
> tableEnv.executeSql("drop temporary function if exists foo");
> } {code}
> !image-2021-11-10-21-04-32-660.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
