from:"Ryu Kobayashi \(JIRA\)"

[jira] [Commented] (HIVE-28309) Task Execution Summary duration is not displayed

2024-06-10 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-28309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853884#comment-17853884
 ] 

Ryu Kobayashi commented on HIVE-28309:
--

Summary:

This issue is caused by HIVE-21520. In this ticket, TezJobMonitor was fixed to 
receive and use 
[perfLogger|https://github.com/apache/hive/pull/4996/files#diff-c507c9686697cd2b2d199e5f3936179b2e7717677f3f314d171b9591123edf4cR120].
 This caused a problem because the RenderStrategy in TezJobMonitor referenced a 
[PerfLogger|https://github.com/apache/hive/pull/5285/files#diff-3a08d8bc36d3363a6cb9e5eb8bfba29d0d0e9b6a896419da66e756e4d6cd03bcL61]
 that was a different object than the one in TezJobMonitor.

> Task Execution Summary duration is not displayed
> 
>
> Key: HIVE-28309
> URL: https://issues.apache.org/jira/browse/HIVE-28309
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
>  Labels: pull-request-available
>
> `DURATION(ms)` is always displayed as 0.
> {code:java}
> Task Execution Summary
> --
>   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   INPUT_RECORDS   
> OUTPUT_RECORDS
> --
>  Map 1  0.00 10,690246  15,000
> 1,231
>  Reducer 2  0.00  2,300  0   1,231
> 0
> --
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-28309) Task Execution Summary duration is not displayed

2024-06-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-28309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-28309 started by Ryu Kobayashi.

> Task Execution Summary duration is not displayed
> 
>
> Key: HIVE-28309
> URL: https://issues.apache.org/jira/browse/HIVE-28309
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
>  Labels: pull-request-available
>
> `DURATION(ms)` is always displayed as 0.
> {code:java}
> Task Execution Summary
> --
>   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   INPUT_RECORDS   
> OUTPUT_RECORDS
> --
>  Map 1  0.00 10,690246  15,000
> 1,231
>  Reducer 2  0.00  2,300  0   1,231
> 0
> --
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-28309) Task Execution Summary duration is not displayed

2024-06-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-28309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-28309:


Assignee: Ryu Kobayashi

> Task Execution Summary duration is not displayed
> 
>
> Key: HIVE-28309
> URL: https://issues.apache.org/jira/browse/HIVE-28309
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
>
> `DURATION(ms)` is always displayed as 0.
> {code:java}
> Task Execution Summary
> --
>   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   INPUT_RECORDS   
> OUTPUT_RECORDS
> --
>  Map 1  0.00 10,690246  15,000
> 1,231
>  Reducer 2  0.00  2,300  0   1,231
> 0
> --
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-28309) Task Execution Summary duration is not displayed

2024-06-07 Thread Ryu Kobayashi (Jira)

Ryu Kobayashi created HIVE-28309:


 Summary: Task Execution Summary duration is not displayed
 Key: HIVE-28309
 URL: https://issues.apache.org/jira/browse/HIVE-28309
 Project: Hive
  Issue Type: Bug
  Components: Clients
Reporter: Ryu Kobayashi


`DURATION(ms)` is always displayed as 0.
{code:java}
Task Execution Summary
--
  VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   INPUT_RECORDS   
OUTPUT_RECORDS
--
 Map 1  0.00 10,690246  15,000  
  1,231
 Reducer 2  0.00  2,300  0   1,231  
  0
--
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-11353) Map env does not reflect in the Local Map Join

2024-04-15 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-11353:
-
Fix Version/s: All Versions
   Resolution: Won't Fix
   Status: Resolved  (was: Patch Available)

> Map env does not reflect in the Local Map Join
> --
>
> Key: HIVE-11353
> URL: https://issues.apache.org/jira/browse/HIVE-11353
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Fix For: All Versions
>
> Attachments: HIVE-11353.1.patch
>
>
> mapreduce.map.env is not reflected when the Local Map Join is ran. Following 
> a sample query:
> {code}
> hive> set mapreduce.map.env=AAA=111,BBB=222,CCC=333;
> hive> select
> >   reflect("java.lang.System", "getenv", "CCC") as CCC,
> >   a.AAA,
> >   b.BBB
> > from (
> >   SELECT
> > reflect("java.lang.System", "getenv", "AAA") as AAA
> >   from
> > foo
> > ) a
> > join (
> >   select
> > reflect("java.lang.System", "getenv", "BBB") as BBB
> >   from
> > foo
> > ) b
> > limit 1;
> Warning: Map Join MAPJOIN[10][bigTable=?] in task 'Stage-3:MAPRED' is a cross 
> product
> Query ID = root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13
> Total jobs = 1
> 15/07/16 01:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> Execution log at: 
> /tmp/root/root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13.log
> 2015-07-16 01:36:47 Starting to launch local task to process map join;
>   maximum memory = 477102080
> 2015-07-16 01:36:48 Dump the side-table for tag: 0 with group count: 1 
> into file: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2015-07-16 01:36:48 Uploaded 1 File to: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
>  (282 bytes)
> 2015-07-16 01:36:48 End of local task; Time Taken: 0.934 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1436962851556_0015, Tracking URL = 
> http://hadoop27:8088/proxy/application_1436962851556_0015/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1436962851556_0015
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2015-07-16 01:36:56,488 Stage-3 map = 0%,  reduce = 0%
> 2015-07-16 01:37:01,656 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.28 
> sec
> MapReduce Total cumulative CPU time: 1 seconds 280 msec
> Ended Job = job_1436962851556_0015
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1   Cumulative CPU: 1.28 sec   HDFS Read: 5428 HDFS 
> Write: 13 SUCCESS
> Total MapReduce CPU Time Spent: 1 seconds 280 msec
> OK
> 333 null222
> Time taken: 19.562 seconds, Fetched: 1 row(s)
> {code}
> The attached patch will include those taken from Hadoop's code.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-11353) Map env does not reflect in the Local Map Join

2024-04-15 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837493#comment-17837493
 ] 

Ryu Kobayashi commented on HIVE-11353:
--

MapReduce has been deprecated and this ticket will be closed.

> Map env does not reflect in the Local Map Join
> --
>
> Key: HIVE-11353
> URL: https://issues.apache.org/jira/browse/HIVE-11353
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-11353.1.patch
>
>
> mapreduce.map.env is not reflected when the Local Map Join is ran. Following 
> a sample query:
> {code}
> hive> set mapreduce.map.env=AAA=111,BBB=222,CCC=333;
> hive> select
> >   reflect("java.lang.System", "getenv", "CCC") as CCC,
> >   a.AAA,
> >   b.BBB
> > from (
> >   SELECT
> > reflect("java.lang.System", "getenv", "AAA") as AAA
> >   from
> > foo
> > ) a
> > join (
> >   select
> > reflect("java.lang.System", "getenv", "BBB") as BBB
> >   from
> > foo
> > ) b
> > limit 1;
> Warning: Map Join MAPJOIN[10][bigTable=?] in task 'Stage-3:MAPRED' is a cross 
> product
> Query ID = root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13
> Total jobs = 1
> 15/07/16 01:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> Execution log at: 
> /tmp/root/root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13.log
> 2015-07-16 01:36:47 Starting to launch local task to process map join;
>   maximum memory = 477102080
> 2015-07-16 01:36:48 Dump the side-table for tag: 0 with group count: 1 
> into file: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2015-07-16 01:36:48 Uploaded 1 File to: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
>  (282 bytes)
> 2015-07-16 01:36:48 End of local task; Time Taken: 0.934 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1436962851556_0015, Tracking URL = 
> http://hadoop27:8088/proxy/application_1436962851556_0015/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1436962851556_0015
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2015-07-16 01:36:56,488 Stage-3 map = 0%,  reduce = 0%
> 2015-07-16 01:37:01,656 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.28 
> sec
> MapReduce Total cumulative CPU time: 1 seconds 280 msec
> Ended Job = job_1436962851556_0015
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1   Cumulative CPU: 1.28 sec   HDFS Read: 5428 HDFS 
> Write: 13 SUCCESS
> Total MapReduce CPU Time Spent: 1 seconds 280 msec
> OK
> 333 null222
> Time taken: 19.562 seconds, Fetched: 1 row(s)
> {code}
> The attached patch will include those taken from Hadoop's code.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2024-03-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi resolved HIVE-27939.
--
Fix Version/s: All Versions
   Resolution: Won't Fix

Since it no longer occurs by setting the 
{{hive.optimize.cte.materialize.full.aggregate.only}} property to false, we 
will close it for now.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Fix For: All Versions
>
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work stopped] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2024-03-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-27939 stopped by Ryu Kobayashi.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-20 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819047#comment-17819047
 ] 

Ryu Kobayashi edited comment on HIVE-27858 at 2/21/24 3:38 AM:
---

[~dkuzmenko] I agree with your opinion. I think it is a no problem to postpone 
this because there is a workaround. Or, I think there is no problem with making 
this ticket void.


was (Author: ryu_kobayashi):
[~dkuzmenko] I agree with your opinion. I think it is a no problem to postpone 
this because there is a workaround.

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-20 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819047#comment-17819047
 ] 

Ryu Kobayashi commented on HIVE-27858:
--

[~dkuzmenko] I agree with your opinion. I think it is a no problem to postpone 
this because there is a workaround.

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-14 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27858:
-
Description: 
OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These did 
not happen in Hive 2 previously.


  was:OOM happens when executing [^query.sql] using a table in [^ddl.sql]. 
These did not happen in Hive 2 previously.


> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-14 Thread Ryu Kobayashi (Jira)



[ https://issues.apache.org/jira/browse/HIVE-27858 ]


Ryu Kobayashi deleted comment on HIVE-27858:
--

was (Author: JIRAUSER300904):
test


> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-14 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817257#comment-17817257
 ] 

Ryu Kobayashi commented on HIVE-27858:
--

test


> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2024-02-14 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817252#comment-17817252
 ] 

Ryu Kobayashi commented on HIVE-27858:
--

[~okumin] and I investigated and found that the root cause was combinatorial 
explosion of execution plans. [^query.sql] has 20 nested CTEs, and after 
expansion, 40,000 Operators, including 4,000 TableScanOperators, are generated. 
The final memory consumption is also huge, since each has a copied String, 
HashMap, and ColumnInfo. The same issue happens even if it reduce the number of 
columns.

heap dump:

!image-2024-02-14-17-30-59-359.png|width=482,height=213!

Operators:

!image-2024-02-14-17-31-32-714.png|width=488,height=21!

We believe that CTE materialization is currently the only effective way. This 
query will succeed if we set `hive.optimize.cte.materialize.threshold` and 
`hive.optimize.cte.materialize.full.aggregate.only=false`. However, due to a 
known bug(HIVE-24167) in CTE materialization, 
`hive.cbo.fallback.strategy=NEVER` will fail. Therefore, we need to disable CBO 
or set `hive.cbo.fallback.strategy=CONSERVATIVE`. And also, re-investigate, we 
found that this issue also happens in Hive 2 when 
`hive.optimize.cte.materialize.threshold=-1`. So, it is considered not to be 
degraded.

In summary, this query will succeed even in Hive 4 if I set the following 
parameters: Therefore, I think that this ticket itself can be closed without 
fix. What do you think?
{code:java}
set hive.optimize.cte.materialize.threshold=3;
set hive.optimize.cte.materialize.full.aggregate.only=false;
set hive.cbo.fallback.strategy=CONSERVATIVE ;{code}

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: okumin
>Priority: Critical
>  Labels: hive-4.0.0-must
> Fix For: Not Applicable
>
> Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, 
> query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26713) StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'

2024-01-22 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809758#comment-17809758
 ] 

Ryu Kobayashi commented on HIVE-26713:
--

Thanks [~zhangbutao] and [~aturoczy] .

> StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'
> ---
>
> Key: HIVE-26713
> URL: https://issues.apache.org/jira/browse/HIVE-26713
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.1.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When LIKE(%xxx%) search is performed, if the character string contains 
> control characters, overflow occurs as follows.
> https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringExpr.java#L345
> {code:java}
> // input[next] == -1
> // shift[input[next] & MAX_BYTE] == 255
> next += shift[input[next] & MAX_BYTE]; {code}
>  
> Stack trace:
> {code:java}
> TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
> attempt_1665986828766_64791_1_00_00_3:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row 
> 2 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:220)
> 3 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:177)
> 4 at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:479)
> 5 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> 6 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> 7 at java.security.AccessController.doPrivileged(Native Method)
> 8 at javax.security.auth.Subject.doAs(Subject.java:422)
> 9 at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> 10at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> 11at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> 12at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> 13at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> 14at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> 15at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> 16at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 17at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18at java.lang.Thread.run(Thread.java:750)
> 19Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row 
> 20at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> 21at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> 22at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> 23at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:194)
> 24... 16 more
> 25Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row 
> 26at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> 27at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> 28... 19 more
> 29Caused by: java.lang.ArrayIndexOutOfBoundsException: 255
> 30at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringExpr$BoyerMooreHorspool.find(StringExpr.java:409)
> 31at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.index(AbstractFilterStringColLikeStringScalar.java:314)
> 32at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.check(AbstractFilterStringColLikeStringScalar.java:307)
> 33at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar.evaluate(AbstractFilterStringColLikeStringScalar.java:115)
> 34at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprOrExpr.evaluate(FilterExprOrExpr.java:183)
> 35at 
>

[jira] [Commented] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2024-01-11 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805902#comment-17805902
 ] 

Ryu Kobayashi commented on HIVE-27370:
--

This issue should still be resolved not yet. I have recreated the PR.

> SUBSTR UDF return '?' against 4-bytes character
> ---
>
> Key: HIVE-27370
> URL: https://issues.apache.org/jira/browse/HIVE-27370
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>
> SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
> branch. Also, this does not happen in vectorized mode, so it is a problem 
> specific to non-vectorized mode. An example is below:
> {code:java}
> -- vectorized mode
> create temporary table foo (str string) stored as orc;
> insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
> SELECT
>   SUBSTR(str, 1, 10) as a1,
>   SUBSTR(str, 10, 3) as a2,
>   SUBSTR(str, -7) as a3,
>   substr(str, 1, 3) as b1,
>   substr(str, 3) as b2,
>   substr(str, -5) as b3
> from foo
> ;
> 安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
> あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
> {code:java}
> -- non-vectorized
> SELECT
>   SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
>   SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
>   SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
>   substr('あa癩いiうu', 1, 3) as b1,
>   substr('あa癩いiうu', 3) as b2,
>   substr('あa癩いiうu', -5) as b3
> ; 
> 安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26713) StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'

2024-01-11 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805444#comment-17805444
 ] 

Ryu Kobayashi commented on HIVE-26713:
--

This issue should still be resolved not yet. I have recreated the PR.

> StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'
> ---
>
> Key: HIVE-26713
> URL: https://issues.apache.org/jira/browse/HIVE-26713
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When LIKE(%xxx%) search is performed, if the character string contains 
> control characters, overflow occurs as follows.
> https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringExpr.java#L345
> {code:java}
> // input[next] == -1
> // shift[input[next] & MAX_BYTE] == 255
> next += shift[input[next] & MAX_BYTE]; {code}
>  
> Stack trace:
> {code:java}
> TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
> attempt_1665986828766_64791_1_00_00_3:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row 
> 2 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:220)
> 3 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:177)
> 4 at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:479)
> 5 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> 6 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> 7 at java.security.AccessController.doPrivileged(Native Method)
> 8 at javax.security.auth.Subject.doAs(Subject.java:422)
> 9 at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> 10at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> 11at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> 12at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> 13at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> 14at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> 15at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> 16at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 17at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18at java.lang.Thread.run(Thread.java:750)
> 19Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row 
> 20at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> 21at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> 22at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> 23at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:194)
> 24... 16 more
> 25Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row 
> 26at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> 27at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> 28... 19 more
> 29Caused by: java.lang.ArrayIndexOutOfBoundsException: 255
> 30at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringExpr$BoyerMooreHorspool.find(StringExpr.java:409)
> 31at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.index(AbstractFilterStringColLikeStringScalar.java:314)
> 32at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.check(AbstractFilterStringColLikeStringScalar.java:307)
> 33at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar.evaluate(AbstractFilterStringColLikeStringScalar.java:115)
> 34at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprOrExpr.evaluate(FilterExprOrExpr.java:183)
> 35at 
>

[jira] [Commented] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2024-01-11 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805441#comment-17805441
 ] 

Ryu Kobayashi commented on HIVE-26339:
--

This issue should still be resolved not yet. I have recreated the PR.

> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
> expressions. Current code also confirmed that the current regular expression 
> pattern cannot be supported by the following LIKE patterns.
> End pattern
> {code:java}
> %abc\%def {code}
> Start pattern
> {code:java}
> abc\%def% {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2024-01-11 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805436#comment-17805436
 ] 

Ryu Kobayashi commented on HIVE-27939:
--

I pushed new PR.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-17 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17797910#comment-17797910
 ] 

Ryu Kobayashi edited comment on HIVE-27939 at 12/17/23 11:15 AM:
-

With PR this problem no longer occurs, but OOM occurs. This issue is probably 
caused by HIVE-27858.


was (Author: ryu_kobayashi):
With PR this problem no longer occurs, but OOM occurs. This issue is probably 
caused by https://issues.apache.org/jira/browse/HIVE-27858.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-17 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17797910#comment-17797910
 ] 

Ryu Kobayashi commented on HIVE-27939:
--

With PR this problem no longer occurs, but OOM occurs. This issue is probably 
caused by https://issues.apache.org/jira/browse/HIVE-27858.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-15 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-27939 started by Ryu Kobayashi.

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-15 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-27939:


Assignee: Ryu Kobayashi

> Many UNION ALL throws SemanticException when trying to remove partition 
> predicates: fail to find child from parent
> --
>
> Key: HIVE-27939
> URL: https://issues.apache.org/jira/browse/HIVE-27939
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: ddl.sql, query.sql
>
>
> I found that the ticket for HIVE-26779 alone does not resolve when using many 
> UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
> [^query.sql], we get a SemanticException similar to HIVE-26779.
> {code:java}
> 23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
> trying to remove partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
> remove partition predicates: fail to find child from parent
>         at 
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
>         at 
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
>         at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
>         at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27939:
-
Description: 
I found that the ticket for HIVE-26779 alone does not resolve when using many 
UNION ALL. When we create a DDL with [^ddl.sql] and execute a query with 
[^query.sql], we get a SemanticException similar to HIVE-26779.
{code:java}
23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
trying to remove partition predicates: fail to find child from parent
org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
remove partition predicates: fail to find child from parent
        at 
org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
        at 
org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
        at 
org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
        at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
        at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
        at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
        at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
        at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
        at 
org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
        at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}

  was:
I found that the ticket for HIVE-26779 alone does not resolve when using many 
UNION ALL. When we create a DDL with ddl.sql and execute a query with 
query.sql, we get a SemanticException similar to HIVE-26779.
{code:java}
23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
trying to remove partition predicates: fail to find child from parent
org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
remove partition predicates: fail to find child from parent
        at 
org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
        at 
org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
        at 
org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
        at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
        at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
        at

[jira] [Created] (HIVE-27939) Many UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent

2023-12-07 Thread Ryu Kobayashi (Jira)

Ryu Kobayashi created HIVE-27939:


 Summary: Many UNION ALL throws SemanticException when trying to 
remove partition predicates: fail to find child from parent
 Key: HIVE-27939
 URL: https://issues.apache.org/jira/browse/HIVE-27939
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 4.0.0-beta-1
Reporter: Ryu Kobayashi
 Attachments: ddl.sql, query.sql

I found that the ticket for HIVE-26779 alone does not resolve when using many 
UNION ALL. When we create a DDL with ddl.sql and execute a query with 
query.sql, we get a SemanticException similar to HIVE-26779.
{code:java}
23/12/07 18:02:01 ERROR ql.Driver: FAILED: SemanticException Exception when 
trying to remove partition predicates: fail to find child from parent
org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to 
remove partition predicates: fail to find child from parent
        at 
org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:809)
        at 
org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:472)
        at 
org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:691)
        at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:301)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13054)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13272)
        at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12628)
        at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:471)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:436)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:430)
        at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
        at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
        at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
        at 
org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:509)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:525)
        at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:807)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2023-12-06 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794009#comment-17794009
 ] 

Ryu Kobayashi commented on HIVE-27858:
--

[~dkuzmenko] No, Hive 3 happen another error.
{code:java}
FAILED: SemanticException [Error 10411]: Grouping sets size cannot be greater 
than 64 {code}

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: John Sherman
>Priority: Critical
>  Labels: hive-4.0.0-must
> Attachments: ddl.sql, query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2023-11-09 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784696#comment-17784696
 ] 

Ryu Kobayashi commented on HIVE-27858:
--

[~glapark]  If it run it without records, I don't know because OOM is currently 
occurring on Hive 4 and master, but it will complete in about 30 seconds on 
Hive 2.
{code:java}
Query Execution Summary
--
OPERATIONDURATION
--
Compile Query  15.46s
Prepare Plan   12.06s
Submit Plan 0.07s
Start DAG   0.21s
Run DAG 0.46s
--
 {code}

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Priority: Major
> Attachments: ddl.sql, query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2023-11-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27858:
-
Description: OOM happens when executing [^query.sql] using a table in 
[^ddl.sql]. These did not happen in Hive 2 previously.  (was: OOM happens when 
executing query.sql using a table in [^ddl.sql]. These did not happen in Hive 2 
previously.)

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Priority: Major
> Attachments: ddl.sql, query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These 
> did not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2023-11-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27858:
-
Description: OOM happens when executing query.sql using a table in 
[^ddl.sql]. These did not happen in Hive 2 previously.  (was: OOM happens when 
executing query.sql using a table in ddl.sql. These did not happen in Hive 2 
previously.)

> OOM happens when selecting many columns and  JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Priority: Major
> Attachments: ddl.sql, query.sql
>
>
> OOM happens when executing query.sql using a table in [^ddl.sql]. These did 
> not happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27858) OOM happens when selecting many columns and JOIN.

2023-11-07 Thread Ryu Kobayashi (Jira)

Ryu Kobayashi created HIVE-27858:


 Summary: OOM happens when selecting many columns and  JOIN.
 Key: HIVE-27858
 URL: https://issues.apache.org/jira/browse/HIVE-27858
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 4.0.0-beta-1
Reporter: Ryu Kobayashi
 Attachments: ddl.sql, query.sql

OOM happens when executing query.sql using a table in ddl.sql. These did not 
happen in Hive 2 previously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-27796) Enable hive.log.explain.output.include.extended even for non hive.server2.webui

2023-10-13 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-27796 started by Ryu Kobayashi.

> Enable hive.log.explain.output.include.extended even for non 
> hive.server2.webui
> ---
>
> Key: HIVE-27796
> URL: https://issues.apache.org/jira/browse/HIVE-27796
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>
> Enable hive.server2.webui even for non 
> hive.log.explain.output.include.extended. If there are many tables, the 
> information will be buried, so make it possible to select.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27796) Enable hive.log.explain.output.include.extended even for non hive.server2.webui

2023-10-13 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27796:
-
Summary: Enable hive.log.explain.output.include.extended even for non 
hive.server2.webui  (was: Enable hive.server2.webui even for non 
hive.log.explain.output.include.extended)

> Enable hive.log.explain.output.include.extended even for non 
> hive.server2.webui
> ---
>
> Key: HIVE-27796
> URL: https://issues.apache.org/jira/browse/HIVE-27796
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Enable hive.server2.webui even for non 
> hive.log.explain.output.include.extended. If there are many tables, the 
> information will be buried, so make it possible to select.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27796) Enable hive.server2.webui even for non hive.log.explain.output.include.extended

2023-10-13 Thread Ryu Kobayashi (Jira)

Ryu Kobayashi created HIVE-27796:


 Summary: Enable hive.server2.webui even for non 
hive.log.explain.output.include.extended
 Key: HIVE-27796
 URL: https://issues.apache.org/jira/browse/HIVE-27796
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Ryu Kobayashi


Enable hive.server2.webui even for non 
hive.log.explain.output.include.extended. If there are many tables, the 
information will be buried, so make it possible to select.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27796) Enable hive.server2.webui even for non hive.log.explain.output.include.extended

2023-10-13 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-27796:


Assignee: Ryu Kobayashi

> Enable hive.server2.webui even for non 
> hive.log.explain.output.include.extended
> ---
>
> Key: HIVE-27796
> URL: https://issues.apache.org/jira/browse/HIVE-27796
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Enable hive.server2.webui even for non 
> hive.log.explain.output.include.extended. If there are many tables, the 
> information will be buried, so make it possible to select.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2023-08-01 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-27370:


Assignee: Ryu Kobayashi  (was: Ryu Kobayashi)

> SUBSTR UDF return '?' against 4-bytes character
> ---
>
> Key: HIVE-27370
> URL: https://issues.apache.org/jira/browse/HIVE-27370
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>
> SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
> branch. Also, this does not happen in vectorized mode, so it is a problem 
> specific to non-vectorized mode. An example is below:
> {code:java}
> -- vectorized mode
> create temporary table foo (str string) stored as orc;
> insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
> SELECT
>   SUBSTR(str, 1, 10) as a1,
>   SUBSTR(str, 10, 3) as a2,
>   SUBSTR(str, -7) as a3,
>   substr(str, 1, 3) as b1,
>   substr(str, 3) as b2,
>   substr(str, -5) as b3
> from foo
> ;
> 安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
> あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
> {code:java}
> -- non-vectorized
> SELECT
>   SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
>   SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
>   SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
>   substr('あa癩いiうu', 1, 3) as b1,
>   substr('あa癩いiうu', 3) as b2,
>   substr('あa癩いiうu', -5) as b3
> ; 
> 安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2023-05-30 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-27370 started by Ryu Kobayashi.

> SUBSTR UDF return '?' against 4-bytes character
> ---
>
> Key: HIVE-27370
> URL: https://issues.apache.org/jira/browse/HIVE-27370
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>
> SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
> branch. Also, this does not happen in vectorized mode, so it is a problem 
> specific to non-vectorized mode. An example is below:
> {code:java}
> -- vectorized mode
> create temporary table foo (str string) stored as orc;
> insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
> SELECT
>   SUBSTR(str, 1, 10) as a1,
>   SUBSTR(str, 10, 3) as a2,
>   SUBSTR(str, -7) as a3,
>   substr(str, 1, 3) as b1,
>   substr(str, 3) as b2,
>   substr(str, -5) as b3
> from foo
> ;
> 安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
> あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
> {code:java}
> -- non-vectorized
> SELECT
>   SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
>   SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
>   SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
>   substr('あa癩いiうu', 1, 3) as b1,
>   substr('あa癩いiうu', 3) as b2,
>   substr('あa癩いiうu', -5) as b3
> ; 
> 安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2023-05-30 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27370:
-
Status: Patch Available  (was: In Progress)

> SUBSTR UDF return '?' against 4-bytes character
> ---
>
> Key: HIVE-27370
> URL: https://issues.apache.org/jira/browse/HIVE-27370
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>
> SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
> branch. Also, this does not happen in vectorized mode, so it is a problem 
> specific to non-vectorized mode. An example is below:
> {code:java}
> -- vectorized mode
> create temporary table foo (str string) stored as orc;
> insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
> SELECT
>   SUBSTR(str, 1, 10) as a1,
>   SUBSTR(str, 10, 3) as a2,
>   SUBSTR(str, -7) as a3,
>   substr(str, 1, 3) as b1,
>   substr(str, 3) as b2,
>   substr(str, -5) as b3
> from foo
> ;
> 安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
> あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
> {code:java}
> -- non-vectorized
> SELECT
>   SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
>   SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
>   SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
>   substr('あa癩いiうu', 1, 3) as b1,
>   substr('あa癩いiうu', 3) as b2,
>   substr('あa癩いiうu', -5) as b3
> ; 
> 安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2023-05-24 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-27370:


Assignee: Ryu Kobayashi

> SUBSTR UDF return '?' against 4-bytes character
> ---
>
> Key: HIVE-27370
> URL: https://issues.apache.org/jira/browse/HIVE-27370
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
> branch. Also, this does not happen in vectorized mode, so it is a problem 
> specific to non-vectorized mode. An example is below:
> {code:java}
> -- vectorized mode
> create temporary table foo (str string) stored as orc;
> insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
> SELECT
>   SUBSTR(str, 1, 10) as a1,
>   SUBSTR(str, 10, 3) as a2,
>   SUBSTR(str, -7) as a3,
>   substr(str, 1, 3) as b1,
>   substr(str, 3) as b2,
>   substr(str, -5) as b3
> from foo
> ;
> 安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
> あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
> {code:java}
> -- non-vectorized
> SELECT
>   SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
>   SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
>   SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
>   substr('あa癩いiうu', 1, 3) as b1,
>   substr('あa癩いiうu', 3) as b2,
>   substr('あa癩いiうu', -5) as b3
> ; 
> 安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27370) SUBSTR UDF return '?' against 4-bytes character

2023-05-24 Thread Ryu Kobayashi (Jira)

Ryu Kobayashi created HIVE-27370:


 Summary: SUBSTR UDF return '?' against 4-bytes character
 Key: HIVE-27370
 URL: https://issues.apache.org/jira/browse/HIVE-27370
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: All Versions
Reporter: Ryu Kobayashi


SUBSTR doesn't seem to support 4-byte characters. This also happens in master 
branch. Also, this does not happen in vectorized mode, so it is a problem 
specific to non-vectorized mode. An example is below:
{code:java}
-- vectorized mode
create temporary table foo (str string) stored as orc;
insert into foo values('安佐町大字久地字野赱4614番地'), ('あa癩いiうu');
SELECT
  SUBSTR(str, 1, 10) as a1,
  SUBSTR(str, 10, 3) as a2,
  SUBSTR(str, -7) as a3,
  substr(str, 1, 3) as b1,
  substr(str, 3) as b2,
  substr(str, -5) as b3
from foo
;
安佐町大字久地字野赱  赱4614番地  安佐町       町大字久地字野赱4614番地     614番地
あa癩             あa癩いiうu        あa癩        癩いiうu    癩いiうu {code}
{code:java}
-- non-vectorized
SELECT
  SUBSTR('安佐町大字久地字野赱4614番地', 1, 10) as a1,
  SUBSTR('安佐町大字久地字野赱4614番地', 10, 3) as a2,
  SUBSTR('安佐町大字久地字野赱4614番地', -7) as a3,
  substr('あa癩いiうu', 1, 3) as b1,
  substr('あa癩いiうu', 3) as b2,
  substr('あa癩いiうu', -5) as b3
; 
安佐町大字久地字野?    �4   ?4614番地     あa?   �いiうu    ?いiうu{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27088) Using MergeJoin and using filters does not work

2023-05-10 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721272#comment-17721272
 ] 

Ryu Kobayashi commented on HIVE-27088:
--

[~kkasa] Sorry, late response. I gave a concrete example.

> Using MergeJoin and using filters does not work
> ---
>
> Key: HIVE-27088
> URL: https://issues.apache.org/jira/browse/HIVE-27088
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
> and OUTER and a filter exists, normal results cannot be obtained.
> For example:
> {code:java}
> -- Data preparation
> create temporary table foo (id bigint, code string) stored as orc;
> create temporary table bar (id bigint, code string) stored as orc;
> create temporary table baz (id bigint) stored as orc;
> INSERT INTO foo values
>   (252073, '01'),
>   (252107, '01'),
>   (252111, '01'),
>   (252112, '01'),
>   (252113, '01'),
>   (252114, '01'),
>   (252071, '01A'),
>   (252072, '01A'),
>   (252116, '01A'),
>   (252117, '01A'),
>   (252118, '01A'),
>   (252119, '01A'),
>   (252120, '01A'),
>   (252076, '06'),
>   (252074, '06A'),
>   (252075, '06A');INSERT INTO bar values
>   (252071, '01'),
>   (252072, '01'),
>   (252073, '01'),
>   (252116, '01'),
>   (252117, '01'),
>   (252071, '01A'),
>   (252072, '01A'),
>   (252073, '01A'),
>   (252116, '01AS'),
>   (252117, '01AS'),
>   (252071, '01B'),
>   (252072, '01B'),
>   (252073, '01B'),
>   (252116, '01BS'),
>   (252117, '01BS'),
>   (252071, '01C'),
>   (252072, '01C'),
>   (252073, '01C7'),
>   (252116, '01CS'),
>   (252117, '01CS'),
>   (252071, '01D'),
>   (252072, '01D'),
>   (252073, '01D'),
>   (252116, '01DS'),
>   (252117, '01DS');INSERT INTO baz values
>   (252071),
>   (252072),
>   (252073),
>   (252074),
>   (252075),
>   (252076),
>   (252107),
>   (252111),
>   (252112),
>   (252113),
>   (252114),
>   (252116),
>   (252117),
>   (252118),
>   (252119),
>   (252120);{code}
> Normal works(set hive.merge.nway.joins=false):
> {code:java}
> hive> set hive.merge.nway.joins=false;
> hive> SELECT
>   a.id,
>   b.code,
>   c.id
> FROM bar AS a
> INNER JOIN foo AS b
> ON a.id = b.id
>   AND (a.code = '01AS' OR b.code = '01BS')
> LEFT OUTER JOIN baz AS c
> ON a.id = c.id;
> OK
> 252116  01A     252116
> 252117  01A     252117 {code}
> Abnormal works(set hive.merge.nway.joins=true):
> {code:java}
> hive> set hive.merge.nway.joins=true;
> hive> SELECT
>   a.id,
>   b.code,
>   c.id
> FROM bar AS a
> INNER JOIN foo AS b
> ON a.id = b.id
>   AND (a.code = '01AS' OR b.code = '01BS')
> LEFT OUTER JOIN baz AS c
> ON a.id = c.id;
> OK 252071  01A     NULL
> 252072  01A     NULL
> 252073  01      NULL
> 252116  01A     NULL
> 252117  01A     NULL
> 252071  01A     NULL
> 252072  01A     NULL
> 252073  01      NULL
> 252116  01A     252116
> 252117  01A     252117
> 252071  01A     NULL
> 252072  01A     NULL
> 252073  01      NULL
> 252116  01A     NULL
> 252117  01A     NULL
> 252071  01A     NULL
> 252072  01A     NULL
> 252073  01      NULL
> 252116  01A     NULL
> 252117  01A     NULL
> 252071  01A     NULL
> 252072  01A     NULL
> 252073  01      NULL
> 252116  01A     NULL
> 252117  01A     NULL   {code}
>  
> I think this is also related to the next ticket: 
> https://issues.apache.org/jira/browse/HIVE-21322



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27088) Using MergeJoin and using filters does not work

2023-05-10 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27088:
-
Description: 
When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
and OUTER and a filter exists, normal results cannot be obtained.

For example:
{code:java}
-- Data preparation
create temporary table foo (id bigint, code string) stored as orc;
create temporary table bar (id bigint, code string) stored as orc;
create temporary table baz (id bigint) stored as orc;
INSERT INTO foo values
  (252073, '01'),
  (252107, '01'),
  (252111, '01'),
  (252112, '01'),
  (252113, '01'),
  (252114, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252116, '01A'),
  (252117, '01A'),
  (252118, '01A'),
  (252119, '01A'),
  (252120, '01A'),
  (252076, '06'),
  (252074, '06A'),
  (252075, '06A');INSERT INTO bar values
  (252071, '01'),
  (252072, '01'),
  (252073, '01'),
  (252116, '01'),
  (252117, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252073, '01A'),
  (252116, '01AS'),
  (252117, '01AS'),
  (252071, '01B'),
  (252072, '01B'),
  (252073, '01B'),
  (252116, '01BS'),
  (252117, '01BS'),
  (252071, '01C'),
  (252072, '01C'),
  (252073, '01C7'),
  (252116, '01CS'),
  (252117, '01CS'),
  (252071, '01D'),
  (252072, '01D'),
  (252073, '01D'),
  (252116, '01DS'),
  (252117, '01DS');INSERT INTO baz values
  (252071),
  (252072),
  (252073),
  (252074),
  (252075),
  (252076),
  (252107),
  (252111),
  (252112),
  (252113),
  (252114),
  (252116),
  (252117),
  (252118),
  (252119),
  (252120);{code}
 

Normal works(set hive.merge.nway.joins=false):
{code:java}
hive> set hive.merge.nway.joins=false;
hive> SELECT
  a.id,
  b.code,
  c.id
FROM bar AS a
INNER JOIN foo AS b
ON a.id = b.id
  AND (a.code = '01AS' OR b.code = '01BS')
LEFT OUTER JOIN baz AS c
ON a.id = c.id;

OK
252116  01A     252116
252117  01A     252117 {code}
Abnormal works(set hive.merge.nway.joins=true):
{code:java}
hive> set hive.merge.nway.joins=true;
hive> SELECT
  a.id,
  b.code,
  c.id
FROM bar AS a
INNER JOIN foo AS b
ON a.id = b.id
  AND (a.code = '01AS' OR b.code = '01BS')
LEFT OUTER JOIN baz AS c
ON a.id = c.id;

OK 252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     252116
252117  01A     252117
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL   {code}
 

I think this is also related to the next ticket: 
https://issues.apache.org/jira/browse/HIVE-21322

  was:
When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
and OUTER and a filter exists, normal results cannot be obtained.

For example:
{code:java}
FROM a
INNER JOIN b
  ON a.id = b.id
AND (a.code = 1 or a.code = 2)
LEFT OUTER JOIN c
  ON a.id = c.id{code}
I think this is also related to the next ticket: 
https://issues.apache.org/jira/browse/HIVE-21322


> Using MergeJoin and using filters does not work
> ---
>
> Key: HIVE-27088
> URL: https://issues.apache.org/jira/browse/HIVE-27088
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
> and OUTER and a filter exists, normal results cannot be obtained.
> For example:
> {code:java}
> -- Data preparation
> create temporary table foo (id bigint, code string) stored as orc;
> create temporary table bar (id bigint, code string) stored as orc;
> create temporary table baz (id bigint) stored as orc;
> INSERT INTO foo values
>   (252073, '01'),
>   (252107, '01'),
>   (252111, '01'),
>

[jira] [Updated] (HIVE-27088) Using MergeJoin and using filters does not work

2023-05-10 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27088:
-
Description: 
When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
and OUTER and a filter exists, normal results cannot be obtained.

For example:
{code:java}
-- Data preparation
create temporary table foo (id bigint, code string) stored as orc;
create temporary table bar (id bigint, code string) stored as orc;
create temporary table baz (id bigint) stored as orc;
INSERT INTO foo values
  (252073, '01'),
  (252107, '01'),
  (252111, '01'),
  (252112, '01'),
  (252113, '01'),
  (252114, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252116, '01A'),
  (252117, '01A'),
  (252118, '01A'),
  (252119, '01A'),
  (252120, '01A'),
  (252076, '06'),
  (252074, '06A'),
  (252075, '06A');INSERT INTO bar values
  (252071, '01'),
  (252072, '01'),
  (252073, '01'),
  (252116, '01'),
  (252117, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252073, '01A'),
  (252116, '01AS'),
  (252117, '01AS'),
  (252071, '01B'),
  (252072, '01B'),
  (252073, '01B'),
  (252116, '01BS'),
  (252117, '01BS'),
  (252071, '01C'),
  (252072, '01C'),
  (252073, '01C7'),
  (252116, '01CS'),
  (252117, '01CS'),
  (252071, '01D'),
  (252072, '01D'),
  (252073, '01D'),
  (252116, '01DS'),
  (252117, '01DS');INSERT INTO baz values
  (252071),
  (252072),
  (252073),
  (252074),
  (252075),
  (252076),
  (252107),
  (252111),
  (252112),
  (252113),
  (252114),
  (252116),
  (252117),
  (252118),
  (252119),
  (252120);{code}
Normal works(set hive.merge.nway.joins=false):
{code:java}
hive> set hive.merge.nway.joins=false;
hive> SELECT
  a.id,
  b.code,
  c.id
FROM bar AS a
INNER JOIN foo AS b
ON a.id = b.id
  AND (a.code = '01AS' OR b.code = '01BS')
LEFT OUTER JOIN baz AS c
ON a.id = c.id;

OK
252116  01A     252116
252117  01A     252117 {code}
Abnormal works(set hive.merge.nway.joins=true):
{code:java}
hive> set hive.merge.nway.joins=true;
hive> SELECT
  a.id,
  b.code,
  c.id
FROM bar AS a
INNER JOIN foo AS b
ON a.id = b.id
  AND (a.code = '01AS' OR b.code = '01BS')
LEFT OUTER JOIN baz AS c
ON a.id = c.id;

OK 252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     252116
252117  01A     252117
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL
252071  01A     NULL
252072  01A     NULL
252073  01      NULL
252116  01A     NULL
252117  01A     NULL   {code}
 

I think this is also related to the next ticket: 
https://issues.apache.org/jira/browse/HIVE-21322

  was:
When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
and OUTER and a filter exists, normal results cannot be obtained.

For example:
{code:java}
-- Data preparation
create temporary table foo (id bigint, code string) stored as orc;
create temporary table bar (id bigint, code string) stored as orc;
create temporary table baz (id bigint) stored as orc;
INSERT INTO foo values
  (252073, '01'),
  (252107, '01'),
  (252111, '01'),
  (252112, '01'),
  (252113, '01'),
  (252114, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252116, '01A'),
  (252117, '01A'),
  (252118, '01A'),
  (252119, '01A'),
  (252120, '01A'),
  (252076, '06'),
  (252074, '06A'),
  (252075, '06A');INSERT INTO bar values
  (252071, '01'),
  (252072, '01'),
  (252073, '01'),
  (252116, '01'),
  (252117, '01'),
  (252071, '01A'),
  (252072, '01A'),
  (252073, '01A'),
  (252116, '01AS'),
  (252117, '01AS'),
  (252071, '01B'),
  (252072, '01B'),
  (252073, '01B'),
  (252116, '01BS'),
  (252117, '01BS'),
  (252071, '01C'),
  (252072, '01C'),
  (252073, '01C7'),
  (252116, '01CS'),
  (252117, '01CS'),
  (252071,

[jira] [Updated] (HIVE-27088) Using MergeJoin and using filters does not work

2023-02-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27088:
-
Assignee: Ryu Kobayashi
  Status: Patch Available  (was: Open)

> Using MergeJoin and using filters does not work
> ---
>
> Key: HIVE-27088
> URL: https://issues.apache.org/jira/browse/HIVE-27088
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
> and OUTER and a filter exists, normal results cannot be obtained.
> For example:
> {code:java}
> FROM a
> INNER JOIN b
>   ON a.id = b.id
> AND (a.code = 1 or a.code = 2)
> LEFT OUTER JOIN c
>   ON a.id = c.id{code}
> I think this is also related to the next ticket: 
> https://issues.apache.org/jira/browse/HIVE-21322



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27088) Using MergeJoin and using filters does not work

2023-02-16 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-27088:
-
Description: 
When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
and OUTER and a filter exists, normal results cannot be obtained.

For example:
{code:java}
FROM a
INNER JOIN b
  ON a.id = b.id
AND (a.code = 1 or a.code = 2)
LEFT OUTER JOIN c
  ON a.id = c.id{code}
I think this is also related to the next ticket: 
https://issues.apache.org/jira/browse/HIVE-21322

  was:When `hive.merge.nway.joins` is set to true and JOIN is performed with 
INNER and OUTER and a filter exists, normal results cannot be obtained.


> Using MergeJoin and using filters does not work
> ---
>
> Key: HIVE-27088
> URL: https://issues.apache.org/jira/browse/HIVE-27088
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Ryu Kobayashi
>Priority: Major
>
> When hive.merge.nway.joins is set to `true` and JOIN is performed with INNER 
> and OUTER and a filter exists, normal results cannot be obtained.
> For example:
> {code:java}
> FROM a
> INNER JOIN b
>   ON a.id = b.id
> AND (a.code = 1 or a.code = 2)
> LEFT OUTER JOIN c
>   ON a.id = c.id{code}
> I think this is also related to the next ticket: 
> https://issues.apache.org/jira/browse/HIVE-21322



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-26713) StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'

2022-11-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-26713 started by Ryu Kobayashi.

> StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'
> ---
>
> Key: HIVE-26713
> URL: https://issues.apache.org/jira/browse/HIVE-26713
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When LIKE(%xxx%) search is performed, if the character string contains 
> control characters, overflow occurs as follows.
> https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringExpr.java#L345
> {code:java}
> // input[next] == -1
> // shift[input[next] & MAX_BYTE] == 255
> next += shift[input[next] & MAX_BYTE]; {code}
>  
> Stack trace:
> {code:java}
> TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
> attempt_1665986828766_64791_1_00_00_3:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row 
> 2 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:220)
> 3 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:177)
> 4 at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:479)
> 5 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> 6 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> 7 at java.security.AccessController.doPrivileged(Native Method)
> 8 at javax.security.auth.Subject.doAs(Subject.java:422)
> 9 at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> 10at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> 11at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> 12at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> 13at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> 14at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> 15at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> 16at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 17at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18at java.lang.Thread.run(Thread.java:750)
> 19Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row 
> 20at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> 21at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> 22at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> 23at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:194)
> 24... 16 more
> 25Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row 
> 26at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> 27at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> 28... 19 more
> 29Caused by: java.lang.ArrayIndexOutOfBoundsException: 255
> 30at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringExpr$BoyerMooreHorspool.find(StringExpr.java:409)
> 31at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.index(AbstractFilterStringColLikeStringScalar.java:314)
> 32at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.check(AbstractFilterStringColLikeStringScalar.java:307)
> 33at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar.evaluate(AbstractFilterStringColLikeStringScalar.java:115)
> 34at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprOrExpr.evaluate(FilterExprOrExpr.java:183)
> 35at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:42)
> 36at 
>

[jira] [Assigned] (HIVE-26713) StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'

2022-11-07 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-26713:


Assignee: Ryu Kobayashi

> StringExpr ArrayIndexOutOfBoundsException with LIKE '%xxx%'
> ---
>
> Key: HIVE-26713
> URL: https://issues.apache.org/jira/browse/HIVE-26713
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: All Versions
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When LIKE(%xxx%) search is performed, if the character string contains 
> control characters, overflow occurs as follows.
> https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringExpr.java#L345
> {code:java}
> // input[next] == -1
> // shift[input[next] & MAX_BYTE] == 255
> next += shift[input[next] & MAX_BYTE]; {code}
>  
> Stack trace:
> {code:java}
> TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
> attempt_1665986828766_64791_1_00_00_3:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row 
> 2 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:220)
> 3 at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:177)
> 4 at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:479)
> 5 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> 6 at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> 7 at java.security.AccessController.doPrivileged(Native Method)
> 8 at javax.security.auth.Subject.doAs(Subject.java:422)
> 9 at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> 10at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> 11at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> 12at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> 13at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> 14at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> 15at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> 16at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 17at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18at java.lang.Thread.run(Thread.java:750)
> 19Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row 
> 20at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> 21at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> 22at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> 23at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:194)
> 24... 16 more
> 25Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row 
> 26at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> 27at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> 28... 19 more
> 29Caused by: java.lang.ArrayIndexOutOfBoundsException: 255
> 30at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringExpr$BoyerMooreHorspool.find(StringExpr.java:409)
> 31at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.index(AbstractFilterStringColLikeStringScalar.java:314)
> 32at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar$MiddleChecker.check(AbstractFilterStringColLikeStringScalar.java:307)
> 33at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.AbstractFilterStringColLikeStringScalar.evaluate(AbstractFilterStringColLikeStringScalar.java:115)
> 34at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprOrExpr.evaluate(FilterExprOrExpr.java:183)
> 35at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:42)
> 36at 
>

[jira] [Work started] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2022-06-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-26339 started by Ryu Kobayashi.

> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
> expressions. Current code also confirmed that the current regular expression 
> pattern cannot be supported by the following LIKE patterns.
> End pattern
> {code:java}
> %abc\%def {code}
> Start pattern
> {code:java}
> abc\%def% {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2022-06-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-26339:
-
Description: 
Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
expressions. Current code also confirmed that the current regular expression 
pattern cannot be supported by the following LIKE patterns.

End pattern
{code:java}
%abc\%def {code}
Start pattern
{code:java}
abc\%def% {code}

  was:
Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
expressions. Current code also confirmed that the current regular expression 
pattern cannot be supported by the following LIKE patterns.

End pattern
{code:java}
%abc\def {code}
Start pattern
{code:java}
abc\def% {code}


> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
> expressions. Current code also confirmed that the current regular expression 
> pattern cannot be supported by the following LIKE patterns.
> End pattern
> {code:java}
> %abc\%def {code}
> Start pattern
> {code:java}
> abc\%def% {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2022-06-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-26339:
-
Description: 
Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
expressions. Current code also confirmed that the current regular expression 
pattern cannot be supported by the following LIKE patterns.

End pattern
{code:java}
%abc\def {code}
Start pattern
{code:java}
abc\def% {code}

  was:
Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
expressions. Current code also confirmed that the current regular expression 
pattern cannot be supported by the following LIKE patterns.

End pattern
```
%abc\def
```

Start pattern
```
abc\def%
```


> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
> expressions. Current code also confirmed that the current regular expression 
> pattern cannot be supported by the following LIKE patterns.
> End pattern
> {code:java}
> %abc\def {code}
> Start pattern
> {code:java}
> abc\def% {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2022-06-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-26339:
-
Description: 
Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
expressions. Current code also confirmed that the current regular expression 
pattern cannot be supported by the following LIKE patterns.

End pattern
```
%abc\def
```

Start pattern
```
abc\def%
```

> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Fixed https://issues.apache.org/jira/browse/HIVE-26047 without using regular 
> expressions. Current code also confirmed that the current regular expression 
> pattern cannot be supported by the following LIKE patterns.
> End pattern
> ```
> %abc\def
> ```
> Start pattern
> ```
> abc\def%
> ```



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Assigned] (HIVE-26339) HIVE-26047 Related LIKE pattern issues

2022-06-17 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-26339:


Assignee: Ryu Kobayashi

> HIVE-26047 Related LIKE pattern issues
> --
>
> Key: HIVE-26339
> URL: https://issues.apache.org/jira/browse/HIVE-26339
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Work started] (HIVE-25043) Support custom UDF in Vectorized mode

2021-04-21 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-25043 started by Ryu Kobayashi.

> Support custom UDF in Vectorized mode
> -
>
> Key: HIVE-25043
> URL: https://issues.apache.org/jira/browse/HIVE-25043
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> hive.vectorized.adaptor.usage.mode=chosen does not allow custom UDFs as far 
> as I can see in the code. So, change it to allow only the specified custom 
> UDF.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-25043) Support custom UDF in Vectorized mode

2021-04-21 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-25043:



> Support custom UDF in Vectorized mode
> -
>
> Key: HIVE-25043
> URL: https://issues.apache.org/jira/browse/HIVE-25043
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> hive.vectorized.adaptor.usage.mode=chosen does not allow custom UDFs as far 
> as I can see in the code. So, change it to allow only the specified custom 
> UDF.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-24765) ClassCastException with AND or OR condition

2021-02-10 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282380#comment-17282380
 ] 

Ryu Kobayashi commented on HIVE-24765:
--

[~kgyrtkirk] It's 3.1.2. However, it backported the patch described in the 
description. And I have CBO enabled. Currently I am creating a PR. Is there 
anything else you care about?

> ClassCastException with AND or OR condition
> ---
>
> Key: HIVE-24765
> URL: https://issues.apache.org/jira/browse/HIVE-24765
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> I previously created the following ticket: 
> https://issues.apache.org/jira/browse/HIVE-11708
> However, it turns out that an error occurs under the following conditions:
> {code:java}
> CREATE TABLE tbl(
>   id int,
>   flg int
> );
> set hive.cbo.enable=true;
> SELECT * FROM tbl
> WHERE id >= 100 AND flg = TRUE;
> java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
>   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
>   at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
>   ... 13 more
> {code}
> I know this is a cast issue as well as the previous issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-24765) ClassCastException with AND or OR condition

2021-02-10 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-24765:
-
Description: 
I previously created the following ticket: 
https://issues.apache.org/jira/browse/HIVE-11708

However, it turns out that an error occurs under the following conditions:
{code:java}
CREATE TABLE tbl(
  id int,
  flg int
);

set hive.cbo.enable=true;

SELECT * FROM tbl
WHERE id >= 100 AND flg = TRUE;

java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
org.apache.hadoop.io.BooleanWritable
  at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
  at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
org.apache.hadoop.io.BooleanWritable
  at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
  at 
org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
  at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
  at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
  at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
  at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426)
  at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
  ... 13 more

{code}
I know this is a cast issue as well as the previous issue.

  was:
I previously created the following ticket: 
https://issues.apache.org/jira/browse/HIVE-11708

However, it turns out that an error occurs under the following conditions:
{code:java}
CREATE TABLE tbl(
  id int,
  flg int
);

SELECT * FROM tbl
WHERE id >= 100 AND flg = TRUE;

java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
org.apache.hadoop.io.BooleanWritable
  at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
  at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
org.apache.hadoop.io.BooleanWritable
  at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
  at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
  at

[jira] [Work started] (HIVE-24765) ClassCastException with AND or OR condition

2021-02-10 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-24765 started by Ryu Kobayashi.

> ClassCastException with AND or OR condition
> ---
>
> Key: HIVE-24765
> URL: https://issues.apache.org/jira/browse/HIVE-24765
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> I previously created the following ticket: 
> https://issues.apache.org/jira/browse/HIVE-11708
> However, it turns out that an error occurs under the following conditions:
> {code:java}
> CREATE TABLE tbl(
>   id int,
>   flg int
> );
> SELECT * FROM tbl
> WHERE id >= 100 AND flg = TRUE;
> java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
>   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
>   at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
>   ... 13 more
> {code}
> I know this is a cast issue as well as the previous issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-24765) ClassCastException with AND or OR condition

2021-02-10 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-24765:



> ClassCastException with AND or OR condition
> ---
>
> Key: HIVE-24765
> URL: https://issues.apache.org/jira/browse/HIVE-24765
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> I previously created the following ticket: 
> https://issues.apache.org/jira/browse/HIVE-11708
> However, it turns out that an error occurs under the following conditions:
> {code:java}
> CREATE TABLE tbl(
>   id int,
>   flg int
> );
> SELECT * FROM tbl
> WHERE id >= 100 AND flg = TRUE;
> java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
>   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
>   at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to 
> org.apache.hadoop.io.BooleanWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>   at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
>   ... 13 more
> {code}
> I know this is a cast issue as well as the previous issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-03 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-24110:
-
Attachment: (was: HIVE-24110.01.patch)

> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-03 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190488#comment-17190488
 ] 

Ryu Kobayashi commented on HIVE-24110:
--

pull request: https://github.com/apache/hive/pull/1468

> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-24110.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-03 Thread Ryu Kobayashi (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190478#comment-17190478
 ] 

Ryu Kobayashi commented on HIVE-24110:
--

[~zabetak] Thanks for your information! I will do.

> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-24110.01.patch
>
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-02 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-24110:
-
Attachment: HIVE-24110.01.patch

> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-24110.01.patch
>
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-02 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-24110:
-
Status: Patch Available  (was: Open)

> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-24110.01.patch
>
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-24110) NullPointerException occurs in some UDFs

2020-09-02 Thread Ryu Kobayashi (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-24110:



> NullPointerException occurs in some UDFs
> 
>
> Key: HIVE-24110
> URL: https://issues.apache.org/jira/browse/HIVE-24110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
>
> Since it refers to a variable that has not been initialized, 
> NullPointerException occurs and the correct error message is not displayed.
>  
> {code:java}
> if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
>   throw new UDFArgumentException(
>   "OCTET_LENGTH only takes primitive types, got " + 
> argumentOI.getTypeName());
> }
> argumentOI = (PrimitiveObjectInspector) arguments[0];
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-27 Thread Ryu Kobayashi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753624#comment-16753624
 ] 

Ryu Kobayashi commented on HIVE-11708:
--

[~ashutoshc] Thanks!

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-11708.01.patch, HIVE-11708.02.patch
>
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21157) Slf4j warning again from 2.X.X

2019-01-24 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-21157:
-
Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

> Slf4j warning again from 2.X.X
> --
>
> Key: HIVE-21157
> URL: https://issues.apache.org/jira/browse/HIVE-21157
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-21157.01.patch
>
>
> The warning of Slf4j has occurred again since 2.0.0:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21157) Slf4j warning again from 2.X.X

2019-01-24 Thread Ryu Kobayashi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751828#comment-16751828
 ] 

Ryu Kobayashi commented on HIVE-21157:
--

[~ashutoshc] I see. I wanted to exclude this slf4j's warning but I will 
invalidate this ticket.

> Slf4j warning again from 2.X.X
> --
>
> Key: HIVE-21157
> URL: https://issues.apache.org/jira/browse/HIVE-21157
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-21157.01.patch
>
>
> The warning of Slf4j has occurred again since 2.0.0:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-24 Thread Ryu Kobayashi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751826#comment-16751826
 ] 

Ryu Kobayashi commented on HIVE-11708:
--

[~ashutoshc] Thanks! I added test.

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-11708.01.patch, HIVE-11708.02.patch
>
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-24 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-11708:
-
Attachment: HIVE-11708.02.patch

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-11708.01.patch, HIVE-11708.02.patch
>
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-23 Thread Ryu Kobayashi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750608#comment-16750608
 ] 

Ryu Kobayashi commented on HIVE-11708:
--

I created the patch.

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-11708.01.patch
>
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-11708:
-
Attachment: HIVE-11708.01.patch
Status: Patch Available  (was: In Progress)

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 0.13.0
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
> Attachments: HIVE-11708.01.patch
>
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-11708 started by Ryu Kobayashi.

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-11708) Logical operators raises ClassCastExceptions with NULL

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-11708:


Assignee: Ryu Kobayashi  (was: Lars Francke)

> Logical operators raises ClassCastExceptions with NULL
> --
>
> Key: HIVE-11708
> URL: https://issues.apache.org/jira/browse/HIVE-11708
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 1.2.1
>Reporter: Satoshi Tagomori
>Assignee: Ryu Kobayashi
>Priority: Major
>
> According to Language Manual UDF, logical operators returns NULL if one of 
> arguments is NULL.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-LogicalOperators
> But query below fails with ClassCastException.
> {code}
> SELECT COUNT(*) AS c
> FROM tbl
> WHERE 1=1 AND NULL
> {code}
> Exception (on 0.13):
> {noformat}
> 15/08/27 08:56:23 ERROR ql.Driver: FAILED: ClassCastException 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:52)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:116)
>   at 
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:231)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:934)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1128)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9716)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9672)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3208)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3005)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8228)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8183)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9015)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9281)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> {noformat}
> I confirmed that Hive 1.2.1 of HDP2.3 Sandbox also raises this exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21157) Slf4j warning again from 2.X.X

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-21157:
-
Attachment: HIVE-21157.01.patch
Status: Patch Available  (was: In Progress)

> Slf4j warning again from 2.X.X
> --
>
> Key: HIVE-21157
> URL: https://issues.apache.org/jira/browse/HIVE-21157
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-21157.01.patch
>
>
> The warning of Slf4j has occurred again since 2.0.0:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-21157) Slf4j warning again from 2.X.X

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21157 started by Ryu Kobayashi.

> Slf4j warning again from 2.X.X
> --
>
> Key: HIVE-21157
> URL: https://issues.apache.org/jira/browse/HIVE-21157
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
>
> The warning of Slf4j has occurred again since 2.0.0:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21157) Slf4j warning again from 2.X.X

2019-01-23 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-21157:



> Slf4j warning again from 2.X.X
> --
>
> Key: HIVE-21157
> URL: https://issues.apache.org/jira/browse/HIVE-21157
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
>
> The warning of Slf4j has occurred again since 2.0.0:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: (was: HIVE-20318.02.patch)

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch, 
> HIVE-20318.03.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.03.patch

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch, 
> HIVE-20318.03.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.02.patch

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch, 
> HIVE-20318.03.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.02.patch

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: (was: HIVE-20318.02.patch)

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-16 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.02.patch

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch, HIVE-20318.02.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi reassigned HIVE-20318:


Assignee: Ryu Kobayashi

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.01.patch
Status: Patch Available  (was: In Progress)

This problem is caused by HIVE-16227. The problem is the case of SELECT instead 
of multi-insert as above.

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.1.1, 3.1.0, 2.3.2, 3.0.0
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Comment: was deleted

(was: This problem is caused by HIVE-16227. The problem is the case of SELECT 
instead of multi-insert as above.)

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-20318 started by Ryu Kobayashi.

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: (was: HIVE-20318.01.patch)

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740112#comment-16740112
 ] 

Ryu Kobayashi commented on HIVE-20318:
--

This problem is caused by HIVE-16227. The problem is the case of SELECT instead 
of multi-insert as above.

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Attachment: HIVE-20318.01.patch

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Assignee: Ryu Kobayashi
>Priority: Blocker
> Attachments: HIVE-20318.01.patch
>
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Affects Version/s: 3.0.0
   3.1.0
   3.1.1

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3
>Reporter: Wang Yan
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view

2019-01-10 Thread Ryu Kobayashi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-20318:
-
Environment: Run on MR, hadoop 2.7.3 and Tez  (was: Run on MR, hadoop 2.7.3)

> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 3.0.0, 2.3.2, 3.1.0, 3.1.1
> Environment: Run on MR, hadoop 2.7.3 and Tez
>Reporter: Wang Yan
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH 
> t1 AS (SELECT 0 AS c1),
> t2 AS (
>   SELECT 0 AS c1
>   FROM (
>   SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
>   )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
>  2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: Hive Runtime Error 
> while closing operators
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>  Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
>  at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
>  ... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-15402) LAG's PRECEDING does not work.

2017-10-30 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi resolved HIVE-15402.
--
Resolution: Won't Fix

> LAG's PRECEDING does not work.
> --
>
> Key: HIVE-15402
> URL: https://issues.apache.org/jira/browse/HIVE-15402
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>
> The syntax in the following manual does not work: 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics#LanguageManualWindowingAndAnalytics-LAGspecifyingalagof3rowsanddefaultvalueof0
> {code}
> SELECT a, LAG(a, 3, 0) OVER (PARTITION BY b ORDER BY C ROWS 3 PRECEDING)
> FROM T;
> {code}
> {code}
> FAILED: SemanticException Failed to breakup Windowing invocations into 
> Groups. At least 1 group must only depend on input columns. Also check for 
> circular dependencies.
> Underlying error: Expecting left window frame boundary for function 
> LAG((tok_table_or_col a), 3, 0) Window 
> Spec=[PartitioningSpec=[partitionColumns=[(tok_table_or_col 
> b)]orderColumns=[(tok_table_or_col c) ASC NULLS_FIRST]]window(start=range(3 
> PRECEDING), end=currentRow)] as LAG_window_0 to be unbounded. Found : 3
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15402) LAG's PRECEDING does not work.

2017-10-30 Thread Ryu Kobayashi (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-15402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226000#comment-16226000
 ] 

Ryu Kobayashi commented on HIVE-15402:
--

(y)

> LAG's PRECEDING does not work.
> --
>
> Key: HIVE-15402
> URL: https://issues.apache.org/jira/browse/HIVE-15402
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>
> The syntax in the following manual does not work: 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics#LanguageManualWindowingAndAnalytics-LAGspecifyingalagof3rowsanddefaultvalueof0
> {code}
> SELECT a, LAG(a, 3, 0) OVER (PARTITION BY b ORDER BY C ROWS 3 PRECEDING)
> FROM T;
> {code}
> {code}
> FAILED: SemanticException Failed to breakup Windowing invocations into 
> Groups. At least 1 group must only depend on input columns. Also check for 
> circular dependencies.
> Underlying error: Expecting left window frame boundary for function 
> LAG((tok_table_or_col a), 3, 0) Window 
> Spec=[PartitioningSpec=[partitionColumns=[(tok_table_or_col 
> b)]orderColumns=[(tok_table_or_col c) ASC NULLS_FIRST]]window(start=range(3 
> PRECEDING), end=currentRow)] as LAG_window_0 to be unbounded. Found : 3
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14481) Remove the comments from the query

2016-08-08 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-14481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-14481:
-
Assignee: Ryu Kobayashi
  Status: Patch Available  (was: Open)

> Remove the comments from the query
> --
>
> Key: HIVE-14481
> URL: https://issues.apache.org/jira/browse/HIVE-14481
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
> Attachments: HIVE-14481.1.patch
>
>
> The ability to delete a comment in CliDriver was created in the next ticket:
> HIVE-1926, HIVE-1953
> However, the following query will result in an error:
> {code}
> -- set abc=def;
> select -- comments;
>   -- comments;
>   replace('12345', '12', '--') -- comments;
> from
>   www_access
> limit 1;
> {code}
> It was to remove all of the comments in order to cope with this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14481) Remove the comments from the query

2016-08-08 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-14481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-14481:
-
Attachment: HIVE-14481.1.patch

> Remove the comments from the query
> --
>
> Key: HIVE-14481
> URL: https://issues.apache.org/jira/browse/HIVE-14481
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
> Attachments: HIVE-14481.1.patch
>
>
> The ability to delete a comment in CliDriver was created in the next ticket:
> HIVE-1926, HIVE-1953
> However, the following query will result in an error:
> {code}
> -- set abc=def;
> select -- comments;
>   -- comments;
>   replace('12345', '12', '--') -- comments;
> from
>   www_access
> limit 1;
> {code}
> It was to remove all of the comments in order to cope with this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-14121) Error occurs when it use the Tez, MapJoin and Vectorization.

2016-06-29 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi resolved HIVE-14121.
--
   Resolution: Won't Fix
Fix Version/s: 2.1.1

> Error occurs when it use the Tez, MapJoin and Vectorization.
> 
>
> Key: HIVE-14121
> URL: https://issues.apache.org/jira/browse/HIVE-14121
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Priority: Critical
> Fix For: 2.1.1
>
>
> This issue is 2.1.0 occurs.
> {code}
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.MapRecordProcessor|: Hit 
> error while closing operators - failing tree
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.TezProcessor|: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:360)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:499)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
>   ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected 
> exception from MapJoinOperator : null
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
>   ... 18 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$ReusableRowContainer.setFromOutput(HybridHashTableContainer.java:922)
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$GetAdaptor.setFromRow(HybridHashTableContainer.java:802)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:339)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:390)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-14121) Error occurs when it use the Tez, MapJoin and Vectorization.

2016-06-29 Thread Ryu Kobayashi (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356369#comment-15356369
 ] 

Ryu Kobayashi edited comment on HIVE-14121 at 6/30/16 4:23 AM:
---

Thanks all!. It worked in the workaround. And, it was also confirmed to work 
with patch. I'll close this ticket.


was (Author: ryu_kobayashi):
Thanks all!. It worked in the workaround.

> Error occurs when it use the Tez, MapJoin and Vectorization.
> 
>
> Key: HIVE-14121
> URL: https://issues.apache.org/jira/browse/HIVE-14121
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Priority: Critical
> Fix For: 2.1.1
>
>
> This issue is 2.1.0 occurs.
> {code}
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.MapRecordProcessor|: Hit 
> error while closing operators - failing tree
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.TezProcessor|: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:360)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:499)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
>   ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected 
> exception from MapJoinOperator : null
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
>   ... 18 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$ReusableRowContainer.setFromOutput(HybridHashTableContainer.java:922)
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$GetAdaptor.setFromRow(HybridHashTableContainer.java:802)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:339)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:390)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14121) Error occurs when it use the Tez, MapJoin and Vectorization.

2016-06-29 Thread Ryu Kobayashi (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356369#comment-15356369
 ] 

Ryu Kobayashi commented on HIVE-14121:
--

Thanks all!. It worked in the workaround.

> Error occurs when it use the Tez, MapJoin and Vectorization.
> 
>
> Key: HIVE-14121
> URL: https://issues.apache.org/jira/browse/HIVE-14121
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Priority: Critical
>
> This issue is 2.1.0 occurs.
> {code}
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.MapRecordProcessor|: Hit 
> error while closing operators - failing tree
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.TezProcessor|: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:360)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"xxx":"yyy"}
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:499)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
>   ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected 
> exception from MapJoinOperator : null
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
>   ... 18 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$ReusableRowContainer.setFromOutput(HybridHashTableContainer.java:922)
>   at 
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$GetAdaptor.setFromRow(HybridHashTableContainer.java:802)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:339)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:390)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14115) Custom FetchFormatter is not supported

2016-06-27 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-14115:
-
Status: Patch Available  (was: Open)

> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and 
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>   
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
>   conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER, 
> DefaultFetchFormatter.class.getName());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14115) Custom FetchFormatter is not supported

2016-06-27 Thread Ryu Kobayashi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryu Kobayashi updated HIVE-14115:
-
Attachment: HIVE-14115.01.patch

> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and 
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>   
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
>   conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER, 
> DefaultFetchFormatter.class.getName());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11353) Map env does not reflect in the Local Map Join

2015-09-09 Thread Ryu Kobayashi (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736366#comment-14736366
 ] 

Ryu Kobayashi commented on HIVE-11353:
--

[~ashutoshc], [~brocknoland] Could you check please?

> Map env does not reflect in the Local Map Join
> --
>
> Key: HIVE-11353
> URL: https://issues.apache.org/jira/browse/HIVE-11353
> Project: Hive
>  Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
> Attachments: HIVE-11353.1.patch
>
>
> mapreduce.map.env is not reflected when the Local Map Join is ran. Following 
> a sample query:
> {code}
> hive> set mapreduce.map.env=AAA=111,BBB=222,CCC=333;
> hive> select
> >   reflect("java.lang.System", "getenv", "CCC") as CCC,
> >   a.AAA,
> >   b.BBB
> > from (
> >   SELECT
> > reflect("java.lang.System", "getenv", "AAA") as AAA
> >   from
> > foo
> > ) a
> > join (
> >   select
> > reflect("java.lang.System", "getenv", "BBB") as BBB
> >   from
> > foo
> > ) b
> > limit 1;
> Warning: Map Join MAPJOIN[10][bigTable=?] in task 'Stage-3:MAPRED' is a cross 
> product
> Query ID = root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13
> Total jobs = 1
> 15/07/16 01:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> Execution log at: 
> /tmp/root/root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13.log
> 2015-07-16 01:36:47 Starting to launch local task to process map join;
>   maximum memory = 477102080
> 2015-07-16 01:36:48 Dump the side-table for tag: 0 with group count: 1 
> into file: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2015-07-16 01:36:48 Uploaded 1 File to: 
> file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
>  (282 bytes)
> 2015-07-16 01:36:48 End of local task; Time Taken: 0.934 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1436962851556_0015, Tracking URL = 
> http://hadoop27:8088/proxy/application_1436962851556_0015/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1436962851556_0015
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2015-07-16 01:36:56,488 Stage-3 map = 0%,  reduce = 0%
> 2015-07-16 01:37:01,656 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.28 
> sec
> MapReduce Total cumulative CPU time: 1 seconds 280 msec
> Ended Job = job_1436962851556_0015
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1   Cumulative CPU: 1.28 sec   HDFS Read: 5428 HDFS 
> Write: 13 SUCCESS
> Total MapReduce CPU Time Spent: 1 seconds 280 msec
> OK
> 333 null222
> Time taken: 19.562 seconds, Fetched: 1 row(s)
> {code}
> The attached patch will include those taken from Hadoop's code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 108 matches

Mail list logo