[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-03-30 Thread Lefty Leverenz (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948703#comment-15948703
 ] 

Lefty Leverenz commented on HIVE-15434:
---

Doc note:  The extract_union function needs to be documented in the wiki for 
release 2.3.0.  It belongs in the Data Types doc and the Operators and UDFs doc.

* [Hive Data Types -- Union Types | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-UnionTypesunionUnionTypes]
* [Hive Operators and UDFs -- Collection Functions | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-CollectionFunctions]

Perhaps there should also be a crossreference here:

* [Hive Operators and UDFs -- Operators on Complex Types | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-OperatorsonComplexTypes]

Added a TODOC2.3 label.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
>  Labels: TODOC2.3
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-03-29 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947471#comment-15947471
 ] 

Ashutosh Chauhan commented on HIVE-15434:
-

Pushed to master and branch-2

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-03-28 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946578#comment-15946578
 ] 

Ashutosh Chauhan commented on HIVE-15434:
-

+1

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-03-25 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942056#comment-15942056
 ] 

Hive QA commented on HIVE-15434:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12843576/HIVE-15434.02.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10534 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
 (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=68)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4375/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4375/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4375/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12843576 - PreCommit-HIVE-Build

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-03-25 Thread Pengcheng Xiong (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941988#comment-15941988
 ] 

Pengcheng Xiong commented on HIVE-15434:


Hello, I am deferring this to Hive 3.0 as we are going to cut the first RC and 
it is not marked as blocker. Please feel free to commit to the branch if this 
can be resolved before the release.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-02-17 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871935#comment-15871935
 ] 

Elliot West commented on HIVE-15434:


[~hagleitn], [~ashutoshc], any chance you can take a quick look at the patch?

Thanks in advance.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-01-17 Thread Amareshwari Sriramadasu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825798#comment-15825798
 ] 

Amareshwari Sriramadasu commented on HIVE-15434:


I had a look at the patch. Seems the new UDF is  all new code and not touching 
any of existing code. And there shouldn't be any issue checking this code in.

+1 from my side. UDF code also look fine.

 [~hagleitn] or [~ashutoshc] Can you help have a look at new UDF if it has 
taken care of practices in UDF writing?

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-01-17 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825731#comment-15825731
 ] 

Elliot West commented on HIVE-15434:


Hello [~namit], [~amareshwari], [~hagleitn], [~ashutoshc]. Apologies for the 
spam, but git suggests that you've all made significant contributions to Hive's 
union feature in the past. Might one of you care to review this?

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-01-16 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824563#comment-15824563
 ] 

Elliot West commented on HIVE-15434:


No problem, thanks for the reply.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2017-01-16 Thread Lars Francke (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824534#comment-15824534
 ] 

Lars Francke commented on HIVE-15434:
-

Thanks [~teabot]. Yeah I think I did document some of this in the past but that 
was more or less a coincidence as I stumbled across the feature when I needed 
it. I'm afraid I currently won't be able to help here.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2016-12-19 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760768#comment-15760768
 ] 

Elliot West commented on HIVE-15434:


Hi [~lars_francke], it looks as though you may have authored the section on the 
wiki regarding the uniontype. Could this issue be relevant to your interests 
and experience?

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2016-12-19 Thread David Maughan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760748#comment-15760748
 ] 

David Maughan commented on HIVE-15434:
--

Hi,

Is anyone able to help me with this ticket? The patch is passing but it
seems other tests unrelated to the patch are failing.

- Dave




> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2016-12-16 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754162#comment-15754162
 ] 

Hive QA commented on HIVE-15434:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12843576/HIVE-15434.02.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10825 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=234)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=128)

[stats13.q,router_join_ppr.q,auto_join13.q,vector_mapjoin_reduce.q,ptf_register_tblfn.q,join_merging.q,union_date_trim.q,groupby3_noskew.q,optimize_nullscan.q,join3.q,join38.q,skewjoinopt1.q,join_alt_syntax.q,groupby_sort_1_23.q,timestamp_udf.q]
TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely 
timed out) (batchId=251)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_sort_array] 
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] 
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=92)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2607/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12843576 - PreCommit-HIVE-Build

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2016-12-16 Thread David Maughan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754021#comment-15754021
 ] 

David Maughan commented on HIVE-15434:
--

These failures seem to be completely unrelated to this patch.

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values

2016-12-15 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751619#comment-15751619
 ] 

Hive QA commented on HIVE-15434:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12843408/HIVE-15434.01.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10837 tests 
executed
*Failed tests:*
{noformat}
TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely 
timed out) (batchId=251)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] 
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=93)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2590/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12843408 - PreCommit-HIVE-Build

> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive 
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete. 
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP 
> BY clauses will fail, and Hive does not define syntax to extract the tag or 
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively 
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to 
> Hive users should at least have the ability to inspect and extract values for 
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the 
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs 
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}} 
> field that matches the tag of the union will be populated with the value. In 
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag 
> number matches then the value is returned, if it does not, then null is 
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)