[jira] [Resolved] (HIVE-22887) MR Job cannot set custom OutputCommitter during job execution.
[ https://issues.apache.org/jira/browse/HIVE-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C resolved HIVE-22887. --- Resolution: Won't Fix > MR Job cannot set custom OutputCommitter during job execution. > -- > > Key: HIVE-22887 > URL: https://issues.apache.org/jira/browse/HIVE-22887 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > MapRedTask set jobs OutputCommitter to NullOutputCommitter. > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.prepareJobOutput(JobConf) > this sets the OutputCommitter to NullOutputCommitter always. > conf.setOutputCommitter(NullOutputCommitter.class); > There are some cases where we need to customize the OutputCommitter. > Like on SUCCESS, they need to write some file or do some custom operation in > theie own OutputCommitter > If someone wants to provide their own OutputCommitter then its not possible > in the current implementation. > Related issues: MR - MAPREDUCE-1802, Hive - HIVE-1355. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23583) Upgrade to ant 1.10.9 due to CVEs
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-23583: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Fixed as part of HIVE-26081 > Upgrade to ant 1.10.9 due to CVEs > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528891#comment-17528891 ] Renukaprasad C commented on HIVE-22845: --- I am not working on it, please anyone can take it further and close the issue. Thank you. > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388261#comment-17388261 ] Renukaprasad C commented on HIVE-22845: --- [~nareshpr] can you please take a look this? > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C reassigned HIVE-22845: - Assignee: (was: Renukaprasad C) > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23129) Cast invalid string to date returns incorrect result
[
https://issues.apache.org/jira/browse/HIVE-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322350#comment-17322350
]
Renukaprasad C commented on HIVE-23129:
---
[~yumwang] Thanks for reporting [~kgyrtkirk] Thanks for the feedback. In my
opinion, this is a feature inherited from JDK. Can you please have a look into
it?
> Cast invalid string to date returns incorrect result
>
>
> Key: HIVE-23129
> URL: https://issues.apache.org/jira/browse/HIVE-23129
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Yuming Wang
>Assignee: wenjun ma
>Priority: Major
>
> {noformat}
> hive> select cast('2020-20-20' as date);
> OK
> 2021-08-20
> Time taken: 4.436 seconds, Fetched: 1 row(s)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23583) Upgrade to ant 1.10.9 due to CVEs
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309965#comment-17309965 ] Renukaprasad C commented on HIVE-23583: --- Hadoop upgraded the version already - HADOOP-17516. We can go with this upgrade. Thanks [~ngangam] [~ychena] for review & quick update. Thanks [~krisden] for the PR. [~ngangam]/[~ychena] Can you have a look into the PR? > Upgrade to ant 1.10.9 due to CVEs > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Kevin Risden >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23927) Cast to Timestamp generates different output for Integer & Float values
[ https://issues.apache.org/jira/browse/HIVE-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176800#comment-17176800 ] Renukaprasad C commented on HIVE-23927: --- Thanks [~jcamachorodriguez] & [~pgaref]. We will do the similar implementation as other integer datatype conversion (As suggested by [~pgaref] -Maybe we should make this configurable as well – as we do in longToTimestamp method) in *PrimitiveObjectInspectorUtils.getTimestamp(Object, PrimitiveObjectInspector, boolean).* > Cast to Timestamp generates different output for Integer & Float values > > > Key: HIVE-23927 > URL: https://issues.apache.org/jira/browse/HIVE-23927 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > Double consider the input value as SECOND and converts into Millis internally. > Whereas, Integer value will be considered as Millis and produce different > output. > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(Object, > PrimitiveObjectInspector, boolean) - Handles Integral & Decimal values > differently. This cause the issue. > 0: jdbc:hive2://localhost:1> select cast(1.204135216E9 as timestamp) > Double2TimeStamp, cast(1204135216 as timestamp) Int2TimeStamp from abc > tablesample(1 rows); > OK > INFO : Compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: > Schema(fieldSchemas:[FieldSchema(name:double2timestamp, type:timestamp, > comment:null), FieldSchema(name:int2timestamp, type:timestamp, > comment:null)], properties:null) > INFO : Completed compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.175 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Completed executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.001 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > ++--+ > |double2timestamp| int2timestamp | > ++--+ > | 2008-02-27 18:00:16.0 | 1970-01-14 22:28:55.216 | > ++--+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23927) Cast to Timestamp generates different output for Integer & Float values
[ https://issues.apache.org/jira/browse/HIVE-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175788#comment-17175788 ] Renukaprasad C commented on HIVE-23927: --- Thanks [~klcopp] [~jcamachorodriguez], Please suggest how to proceed with this compatibility issue? Thank you. > Cast to Timestamp generates different output for Integer & Float values > > > Key: HIVE-23927 > URL: https://issues.apache.org/jira/browse/HIVE-23927 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > Double consider the input value as SECOND and converts into Millis internally. > Whereas, Integer value will be considered as Millis and produce different > output. > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(Object, > PrimitiveObjectInspector, boolean) - Handles Integral & Decimal values > differently. This cause the issue. > 0: jdbc:hive2://localhost:1> select cast(1.204135216E9 as timestamp) > Double2TimeStamp, cast(1204135216 as timestamp) Int2TimeStamp from abc > tablesample(1 rows); > OK > INFO : Compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: > Schema(fieldSchemas:[FieldSchema(name:double2timestamp, type:timestamp, > comment:null), FieldSchema(name:int2timestamp, type:timestamp, > comment:null)], properties:null) > INFO : Completed compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.175 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Completed executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.001 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > ++--+ > |double2timestamp| int2timestamp | > ++--+ > | 2008-02-27 18:00:16.0 | 1970-01-14 22:28:55.216 | > ++--+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22932) Unable to kill Beeline with Ctrl+C
[ https://issues.apache.org/jira/browse/HIVE-22932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175566#comment-17175566 ] Renukaprasad C commented on HIVE-22932: --- Sure. Thank you [~wenjunma003]. > Unable to kill Beeline with Ctrl+C > -- > > Key: HIVE-22932 > URL: https://issues.apache.org/jira/browse/HIVE-22932 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: wenjun ma >Priority: Blocker > > Stopped the server and tried to stop the beeline console with "Ctrl+C". But > it unable to kill the process & gets process gets hanged. > Read call got blocked. > Attached the thread dump. > 0: jdbc:hive2://localhost:1> show tables; > Unknown HS2 problem when communicating with Thrift server. > Error: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe (state=08S01,code=0) > 0: jdbc:hive2://localhost:1> Interrupting... Please be patient this may > take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > 2020-02-26 17:40:42 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode): > "NonBlockingInputStreamThread" #16 daemon prio=5 os_prio=0 > tid=0x7f0318c10800 nid=0x258c in Object.wait() [0x7f031c193000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #11 daemon prio=9 os_prio=0 tid=0x7f032006c000 > nid=0x257b runnable [0x] >java.lang.Thread.State: RUNNABLE > "C1 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x7f0320060800 > nid=0x257a waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x7f0320056000 > nid=0x2579 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x7f0320054000 > nid=0x2578 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x7f0320051000 > nid=0x2577 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x7f032004f000 > nid=0x2576 runnable [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0 > tid=0x7f032004b800 nid=0x2575 runnable [0x] >java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7f0320035800 > nid=0x2574 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f0320003800 nid=0x2572 in > Object.wait() [0x7f0324b1c000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe930770> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0xfe930770> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x006a6000 > nid=0x2571 in Object.wait() [0x7f0324c1d000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) >
[jira] [Commented] (HIVE-23927) Cast to Timestamp generates different output for Integer & Float values
[ https://issues.apache.org/jira/browse/HIVE-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175458#comment-17175458 ] Renukaprasad C commented on HIVE-23927: --- [~gopalv] Shall we change the input unit to Millis for double datatype as well? Changing this may break the compatibility for existing users. Please suggest on this. Thank you. > Cast to Timestamp generates different output for Integer & Float values > > > Key: HIVE-23927 > URL: https://issues.apache.org/jira/browse/HIVE-23927 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > Double consider the input value as SECOND and converts into Millis internally. > Whereas, Integer value will be considered as Millis and produce different > output. > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(Object, > PrimitiveObjectInspector, boolean) - Handles Integral & Decimal values > differently. This cause the issue. > 0: jdbc:hive2://localhost:1> select cast(1.204135216E9 as timestamp) > Double2TimeStamp, cast(1204135216 as timestamp) Int2TimeStamp from abc > tablesample(1 rows); > OK > INFO : Compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: > Schema(fieldSchemas:[FieldSchema(name:double2timestamp, type:timestamp, > comment:null), FieldSchema(name:int2timestamp, type:timestamp, > comment:null)], properties:null) > INFO : Completed compiling > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.175 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14): > select cast(1.204135216E9 as timestamp) Double2TimeStamp, cast(1204135216 as > timestamp) Int2TimeStamp from abc tablesample(1 rows) > INFO : Completed executing > command(queryId=renu_20200724140642_70132390-ee12-4214-a2ca-a7e10556fc14); > Time taken: 0.001 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > ++--+ > |double2timestamp| int2timestamp | > ++--+ > | 2008-02-27 18:00:16.0 | 1970-01-14 22:28:55.216 | > ++--+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18454) Incorrect rownum estimation in joins
[
https://issues.apache.org/jira/browse/HIVE-18454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175446#comment-17175446
]
Renukaprasad C commented on HIVE-18454:
---
[~kgyrtkirk],
I tried to simulate the issue with the provided queries. but i couldnt on
branch - 3.1.0. Is there any specific scenario or it got handled as part of
some other issue already? Thanks in advance.
> Incorrect rownum estimation in joins
>
>
> Key: HIVE-18454
> URL: https://issues.apache.org/jira/browse/HIVE-18454
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Priority: Major
>
> row counts seems to be off the charts...sometimes ~12 rows estimated when
> the table has only 10 rows
> {code:java}
> create table s (x int);
> insert into s values
> (1),(2),(3),(4),(5),
> (6),(7),(8),(9),(10);
> create table tu(id_uv int,id_uw int,u int);
> create table tv(id_uv int,v int);
> create table tw(id_uw int,w int);
> from s
> insert overwrite table tu
> select x,x,x
> where x<=6 or x=10
> insert overwrite table tv
> select x,x
> where x<=3 or x=10
> insert overwrite table tw
> select x,x
> ;
> set hive.explain.user=true;
> explain analyze
> select sum(u*v*w) from tu
> join tv on (tu.id_uv=tv.id_uv)
> join tw on (tu.id_uw=tw.id_uw)
> where w>9 and u>1 and v>3;
> desc formatted tv;
> {code}
> explain analyze output:
> {code:java}
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Map 1 <- Map 3 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE) |
> | Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Stage-1|
> | Reducer 2|
> | File Output Operator [FS_21] |
> | Group By Operator [GBY_19] (rows=1/1 width=8) |
> | Output:["_col0"],aggregations:["sum(VALUE._col0)"] |
> | <-Map 1 [CUSTOM_SIMPLE_EDGE] |
> | PARTITION_ONLY_SHUFFLE [RS_18] |
> | Group By Operator [GBY_17] (rows=1/1 width=8) |
> | Output:["_col0"],aggregations:["sum(_col0)"] |
> | Select Operator [SEL_15] (rows=48400/1 width=5) |
> | Output:["_col0"] |
> | Map Join Operator [MAPJOIN_31] (rows=48400/1 width=5) |
> |
> Conds:MAPJOIN_30._col1=RS_13._col0(Inner),HybridGraceHashJoin:true,Output:["_col2","_col4","_col6"]
> |
> | <-Map 4 [BROADCAST_EDGE] |
> | BROADCAST [RS_13]|
> | PartitionCols:_col0|
> | Select Operator [SEL_8] (rows=3/1 width=3) |
> | Output:["_col0","_col1"] |
> | Filter Operator [FIL_29] (rows=3/1 width=3) |
> | predicate:((w > 9) and id_uw is not null) |
> | TableScan [TS_6] (rows=10/10 width=3) |
> |
> default@tw,tw,Tbl:COMPLETE,Col:NONE,Output:["id_uw","w"] |
> | <-Map Join Operator [MAPJOIN_30] (rows=44000/1 width=5) |
> |
> Conds:SEL_2._col0=RS_10._col0(Inner),HybridGraceHashJoin:true,Output:["_col1","_col2","_col4"]
> |
> | <-Map 3 [BROADCAST_EDGE] |
> | BROADCAST [RS_10] |
> | PartitionCols:_col0 |
> | Select Operator [SEL_5] (rows=1632/1 width=3) |
> | Output:["_col0","_col1"] |
> | Filter Operator [FIL_28] (rows=1632/1 width=3) |
> | predicate:((v > 3) and id_uv is not null) |
> | TableScan [TS_3] (rows=4898/4 width=3) |
> |
> default@tv,tv,Tbl:COMPLETE,Col:NONE,Output:["id_uv","v"] |
> | <-Select Operator [SEL_2] (rows=4/6 width=5) |
> | Output:["_col0","_col1","_col2"] |
> | Filter Operator [FIL_27] (rows=4/6 width=5) |
> | predicate:((u > 1) and id_uv is not null and id_uw
> is not null) |
> | TableScan [TS_0] (rows=12/7 width=5) |
> |
> default@tu,tu,Tbl:COMPLETE,Col:NONE,Output:["id_uv","id_uw","u"] |
> {code}
--
This message was sent by Atlassian
[jira] [Assigned] (HIVE-22887) MR Job cannot set custom OutputCommitter during job execution.
[ https://issues.apache.org/jira/browse/HIVE-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C reassigned HIVE-22887: - Assignee: (was: Renukaprasad C) > MR Job cannot set custom OutputCommitter during job execution. > -- > > Key: HIVE-22887 > URL: https://issues.apache.org/jira/browse/HIVE-22887 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > MapRedTask set jobs OutputCommitter to NullOutputCommitter. > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.prepareJobOutput(JobConf) > this sets the OutputCommitter to NullOutputCommitter always. > conf.setOutputCommitter(NullOutputCommitter.class); > There are some cases where we need to customize the OutputCommitter. > Like on SUCCESS, they need to write some file or do some custom operation in > theie own OutputCommitter > If someone wants to provide their own OutputCommitter then its not possible > in the current implementation. > Related issues: MR - MAPREDUCE-1802, Hive - HIVE-1355. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22932) Unable to kill Beeline with Ctrl+C
[ https://issues.apache.org/jira/browse/HIVE-22932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141490#comment-17141490 ] Renukaprasad C commented on HIVE-22932: --- Thanks [~bianqi] & [~zhongjiajie], "your thrift service is not running or your thift service's port cannot communication with beeline." -- Yes, I was stopping both client and servers, during that time i got this issue. I should have tried !q, hope this would have stopped the client process. Since the scenario cannot reproduce again, i cannot test. May be we can mark it as invalid and close. > Unable to kill Beeline with Ctrl+C > -- > > Key: HIVE-22932 > URL: https://issues.apache.org/jira/browse/HIVE-22932 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Blocker > > Stopped the server and tried to stop the beeline console with "Ctrl+C". But > it unable to kill the process & gets process gets hanged. > Read call got blocked. > Attached the thread dump. > 0: jdbc:hive2://localhost:1> show tables; > Unknown HS2 problem when communicating with Thrift server. > Error: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe (state=08S01,code=0) > 0: jdbc:hive2://localhost:1> Interrupting... Please be patient this may > take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > 2020-02-26 17:40:42 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode): > "NonBlockingInputStreamThread" #16 daemon prio=5 os_prio=0 > tid=0x7f0318c10800 nid=0x258c in Object.wait() [0x7f031c193000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #11 daemon prio=9 os_prio=0 tid=0x7f032006c000 > nid=0x257b runnable [0x] >java.lang.Thread.State: RUNNABLE > "C1 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x7f0320060800 > nid=0x257a waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x7f0320056000 > nid=0x2579 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x7f0320054000 > nid=0x2578 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x7f0320051000 > nid=0x2577 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x7f032004f000 > nid=0x2576 runnable [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0 > tid=0x7f032004b800 nid=0x2575 runnable [0x] >java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7f0320035800 > nid=0x2574 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f0320003800 nid=0x2572 in > Object.wait() [0x7f0324b1c000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe930770> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0xfe930770> (a java.lang.ref.ReferenceQueue$Lock) > at
[jira] [Commented] (HIVE-23583) Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by updating to latest ANT
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141482#comment-17141482 ] Renukaprasad C commented on HIVE-23583: --- Test failures are not related to the ant version chagnes, can we trigger the tests again? Also, i have run the test cases locally with version upgrade & both tests passed. > Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by > updating to latest ANT > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23583) Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by updating to latest ANT
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-23583: -- Attachment: (was: HIVE-23583.01.patch) > Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by > updating to latest ANT > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23583) Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by updating to latest ANT
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-23583: -- Attachment: HIVE-23583.01.patch Status: Patch Available (was: Open) > Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by > updating to latest ANT > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23583) Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by updating to latest ANT
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-23583: -- Attachment: HIVE-23583.01.patch > Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by > updating to latest ANT > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23583.01.patch > > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23583) Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by updating to latest ANT
[ https://issues.apache.org/jira/browse/HIVE-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C reassigned HIVE-23583: - > Fix CVE-2020-1945: Apache Ant insecure temporary file vulnerability by > updating to latest ANT > - > > Key: HIVE-23583 > URL: https://issues.apache.org/jira/browse/HIVE-23583 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Fix For: 4.0.0 > > > Update ANT to fix: > CVE-2020-1945: Apache Ant insecure temporary file vulnerability > Severity: Medium > Vendor: > The Apache Software Foundation > Versions Affected: > Apache Ant 1.1 to 1.9.14 and 1.10.0 to 1.10.7 > Description: > Apache Ant uses the default temporary directory identified by the Java > system property java.io.tmpdir for several tasks and may thus leak > sensitive information. The fixcrlf and replaceregexp tasks also copy > files from the temporary directory back into the build tree allowing an > attacker to inject modified source files into the build process. > Mitigation: > Ant users of versions 1.1 to 1.9.14 and 1.10.0 to 1.10.7 should set the > java.io.tmpdir system property to point to a directory only readable and > writable by the current user prior to running Ant. > Users of versions 1.9.15 and 1.10.8 can use the Ant property ant.tmpfile > instead. Users of Ant 1.10.8 can rely on Ant protecting the temporary > files if the underlying filesystem allows it, but we still recommend > using a private temporary directory instead. > References: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-1945 > https://nvd.nist.gov/vuln/detail/CVE-2020-1945 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23457) Hive Incorrect result with subquery while optimizer misses the aggregation stage
[
https://issues.apache.org/jira/browse/HIVE-23457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115149#comment-17115149
]
Renukaprasad C commented on HIVE-23457:
---
[~Rajkumar Singh]
Limit is mandatory in Subqueries (HIVE-6348). So the ORDERY_BY is ommitted and
continue to execute the query.
In the console WARN log been dispalayed - WARNING: Order/Sort by without limit
in sub query or view [tmp] is removed, as it's pointless and bad for
performance.
> Hive Incorrect result with subquery while optimizer misses the aggregation
> stage
> -
>
> Key: HIVE-23457
> URL: https://issues.apache.org/jira/browse/HIVE-23457
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Priority: Critical
>
> Steps to Repro:
> {code:java}
> create table abc (id int);
> insert into table abc values (1),(2),(3),(4),(5),(6);
> select * from abc order by id desc
> 6
> 5
> 4
> 3
> 2
> 1
> select `id` from (select * from abc order by id desc ) as tmp;
> 1
> 2
> 3
> 4
> 5
> 6
>
> {code}
> looking at the query plan it seems while using the subquery optimizer missed
> the aggregation stage, I cant see any reduce stage.
> {code:java}
> set hive.query.results.cache.enabled=false;
> explain select * from abc order by id desc;
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE) |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Stage-1|
> | Reducer 2 vectorized |
> | File Output Operator [FS_8] |
> | Select Operator [SEL_7] (rows=6 width=4) |
> | Output:["_col0"] |
> | <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_6] |
> | Select Operator [SEL_5] (rows=6 width=4) |
> | Output:["_col0"] |
> | TableScan [TS_0] (rows=6 width=4)|
> | default@abc,abc, ACID
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["id"] |
> ||
> ++
> explain select `id` from (select * from abc order by id desc ) as tmp;
> +--+
> | Explain|
> +--+
> | Plan optimized by CBO. |
> | |
> | Stage-0 |
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1] |
> | Output:["_col0"] |
> | TableScan [TS_0] |
> | Output:["id"]|
> | |
> +--+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22633) GroupByOperator may throw NullPointerException when setting data skew optimization parameters
[
https://issues.apache.org/jira/browse/HIVE-22633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101300#comment-17101300
]
Renukaprasad C commented on HIVE-22633:
---
[~zhangbutao] are you working on this issue?
> GroupByOperator may throw NullPointerException when setting data skew
> optimization parameters
> -
>
> Key: HIVE-22633
> URL: https://issues.apache.org/jira/browse/HIVE-22633
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0, 3.1.1
>Reporter: zhangbutao
>Priority: Major
>
> if hive.map.aggr and hive.groupby.skewindata set true，exception will be
> thrown.
> step to repro:
> 1. create table:
> set hive.map.aggr=true;
> set hive.groupby.skewindata=true;
> create table test1 (id1 bigint);
> create table test2 (id2 bigint) partitioned by(dt2 string);
> insert into test2 partition(dt2='2020') select a.id1 from test1 a group by
> a.id1;
> 2.NullPointerException:
> {code:java}
> ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) :
> attempt_1585641455670_0001_2_03_00_2:java.lang.RuntimeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.init(GenericUDAFComputeStats.java:373)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:373)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23129) Cast invalid string to date returns incorrect result
[
https://issues.apache.org/jira/browse/HIVE-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073624#comment-17073624
]
Renukaprasad C commented on HIVE-23129:
---
java.time.LocalDate.parse(CharSequence, DateTimeFormatter)
This Java API is being used to convert the date, this internally add the excess
months (>12) as year and remaing months gets converted (months % 12).
This same applies to date also.
LocalDate.parse("2020-1-36", PARSE_FORMATTER);
Results to - 2020-02-05
If required to handle, date validation should be done from hive side.
> Cast invalid string to date returns incorrect result
>
>
> Key: HIVE-23129
> URL: https://issues.apache.org/jira/browse/HIVE-23129
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Yuming Wang
>Priority: Major
>
> {noformat}
> hive> select cast('2020-20-20' as date);
> OK
> 2021-08-20
> Time taken: 4.436 seconds, Fetched: 1 row(s)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23115) SQL with "WITH" clause not suppoting Uppercase table name while creating Views
[
https://issues.apache.org/jira/browse/HIVE-23115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072026#comment-17072026
]
Renukaprasad C commented on HIVE-23115:
---
[~venkatadilip], i run the query with latest hive release, i didnt face any
issue. Did you verify with higher version?
> SQL with "WITH" clause not suppoting Uppercase table name while creating Views
> --
>
> Key: HIVE-23115
> URL: https://issues.apache.org/jira/browse/HIVE-23115
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.1
>Reporter: Dilip Kumar
>Priority: Minor
>
> *Beeline version 2.1.1-mapr-1912*
> SQL with "WITH" clause not suppoting Uppercase table name while creating Views
> Below query creates the VIEW successfully,
> CREATE VIEW default.TEST_VIEW as
> with TEMP_TABLE as (select col1,col2 from table1)
> SELECT col1 from TEMP_TABLE limit 1;
> but when I execute select query on the view throws below error
> {color:#FF}Error: Error while compiling statement: FAILED:
> SemanticException Line 3:90 Table not found 'TEMP_TABLE' in definition of
> VIEW TEST_VIEW{color}
>
> When you replace uppercase TEMP_TABLE to lower case temp_table and query the
> Vew-TEST_VIEW it works,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22887) MR Job cannot set custom OutputCommitter during job execution.
[ https://issues.apache.org/jira/browse/HIVE-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037688#comment-17037688 ] Renukaprasad C commented on HIVE-22887: --- [~gopalv] Thank you so much for clarification. Shall we consider for this enhancement for handling special case? If required, we can provide configurable OutputCommitter. > MR Job cannot set custom OutputCommitter during job execution. > -- > > Key: HIVE-22887 > URL: https://issues.apache.org/jira/browse/HIVE-22887 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > > MapRedTask set jobs OutputCommitter to NullOutputCommitter. > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.prepareJobOutput(JobConf) > this sets the OutputCommitter to NullOutputCommitter always. > conf.setOutputCommitter(NullOutputCommitter.class); > There are some cases where we need to customize the OutputCommitter. > Like on SUCCESS, they need to write some file or do some custom operation in > theie own OutputCommitter > If someone wants to provide their own OutputCommitter then its not possible > in the current implementation. > Related issues: MR - MAPREDUCE-1802, Hive - HIVE-1355. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22887) MR Job cannot set custom OutputCommitter during job execution.
[ https://issues.apache.org/jira/browse/HIVE-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C reassigned HIVE-22887: - > MR Job cannot set custom OutputCommitter during job execution. > -- > > Key: HIVE-22887 > URL: https://issues.apache.org/jira/browse/HIVE-22887 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > > MapRedTask set jobs OutputCommitter to NullOutputCommitter. > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.prepareJobOutput(JobConf) > this sets the OutputCommitter to NullOutputCommitter always. > conf.setOutputCommitter(NullOutputCommitter.class); > There are some cases where we need to customize the OutputCommitter. > Like on SUCCESS, they need to write some file or do some custom operation in > theie own OutputCommitter > If someone wants to provide their own OutputCommitter then its not possible > in the current implementation. > Related issues: MR - MAPREDUCE-1802, Hive - HIVE-1355. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032208#comment-17032208 ] Renukaprasad C edited comment on HIVE-22845 at 2/7/20 8:52 AM: --- Replaced '\' with ' ' from user variable to avoid the special character conflicts. was (Author: prasad-acit): Replaced '/' with ' ' from user variable to avoid the special character conflicts. > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032208#comment-17032208 ] Renukaprasad C edited comment on HIVE-22845 at 2/7/20 8:51 AM: --- Replaced '/' with ' ' from user variable to avoid the special character conflicts. was (Author: prasad-acit): Replaced '/' with ' ' from user variable tip avoid the special character conflicts. > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-22845: -- Attachment: HIVE-22845.1.patch Status: Patch Available (was: Open) Replaced '/' with ' ' from user variable tip avoid the special character conflicts. > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-22845: -- Attachment: (was: HIVE-22845.1.patch) > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C updated HIVE-22845: -- Attachment: HIVE-22845.1.patch > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HIVE-22845.1.patch > > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22845) Code compilation fails on Windows when user name begins with 'N' or 'R'
[ https://issues.apache.org/jira/browse/HIVE-22845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renukaprasad C reassigned HIVE-22845: - > Code compilation fails on Windows when user name begins with 'N' or 'R' > --- > > Key: HIVE-22845 > URL: https://issues.apache.org/jira/browse/HIVE-22845 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > > package-info.java is generated from the script - saveVersion.sh. > Generated contetnt is as follows. > /* > * Generated by saveVersion.sh > */ > @MetastoreVersionAnnotation(version="4.0.0-SNAPSHOT", shortVersion="4.0.0", > revision="Unknown", branch="Unknown", > user="India > enu", date="Thu Feb 6 17:12:46 IST 2020", > url="file:///cygdrive/f/hive/code/os/standalone-metastore", > srcChecksum="88f3dc17f75ffda6176faa649593b54e") > package org.apache.hadoop.hive.metastore.annotation; > When user name begings with 'R' (India\Renu), then it get parsed as Carriage > Return in between the string and fails compile. Same issue exists with name > begins with 'N' also. > Whereas compilation is fine with any other user name doesnt contain \r or \n. -- This message was sent by Atlassian Jira (v8.3.4#803005)
