[jira] [Updated] (HIVE-18541) Secure HS2 web UI with PAM
[ https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-18541: Attachment: HIVE-18541.2.patch > Secure HS2 web UI with PAM > -- > > Key: HIVE-18541 > URL: https://issues.apache.org/jira/browse/HIVE-18541 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch > > > Secure HS2 web UI with PAM. Add two new properties > * hive.server2.webui.use.pam > * Default value: false > * Description: If true, the HiveServer2 WebUI will be secured with PAM > * hive.server2.webui.pam.authenticator > * Default value: org.apache.hive.http.security.PamAuthenticator > * Description: Class for PAM authentication -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18541) Secure HS2 web UI with PAM
[ https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-18541: Status: In Progress (was: Patch Available) > Secure HS2 web UI with PAM > -- > > Key: HIVE-18541 > URL: https://issues.apache.org/jira/browse/HIVE-18541 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch > > > Secure HS2 web UI with PAM. Add two new properties > * hive.server2.webui.use.pam > * Default value: false > * Description: If true, the HiveServer2 WebUI will be secured with PAM > * hive.server2.webui.pam.authenticator > * Default value: org.apache.hive.http.security.PamAuthenticator > * Description: Class for PAM authentication -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18541) Secure HS2 web UI with PAM
[ https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-18541: Status: Patch Available (was: In Progress) > Secure HS2 web UI with PAM > -- > > Key: HIVE-18541 > URL: https://issues.apache.org/jira/browse/HIVE-18541 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch > > > Secure HS2 web UI with PAM. Add two new properties > * hive.server2.webui.use.pam > * Default value: false > * Description: If true, the HiveServer2 WebUI will be secured with PAM > * hive.server2.webui.pam.authenticator > * Default value: org.apache.hive.http.security.PamAuthenticator > * Description: Class for PAM authentication -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[
https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18378:
Attachment: HIVE-18378.1.patch
> Explain plan should show if a Map/Reduce Work is being cached
> -
>
> Key: HIVE-18378
> URL: https://issues.apache.org/jira/browse/HIVE-18378
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18378.1.patch
>
>
> It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}}
> objects are being cached by Spark.
> The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark
> cache-ing, so we should be able to modify it so that it displays if a work
> object will be cached or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18648) Investigate SMB related test failures in llap_smb and auto_sortmerge_join_2
[ https://issues.apache.org/jira/browse/HIVE-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-18648: - > Investigate SMB related test failures in llap_smb and auto_sortmerge_join_2 > --- > > Key: HIVE-18648 > URL: https://issues.apache.org/jira/browse/HIVE-18648 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > These tests fail in ptest. Need to root cause and fix or triage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.16.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.13.patch, > HIVE-18350.14.patch, HIVE-18350.15.patch, HIVE-18350.16.patch, > HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch, > HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch, > HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: (was: HIVE-18350.16.patch) > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.13.patch, > HIVE-18350.14.patch, HIVE-18350.15.patch, HIVE-18350.16.patch, > HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch, > HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch, > HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18647) Cannot create table: Unknown column 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
[
https://issues.apache.org/jira/browse/HIVE-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez resolved HIVE-18647.
Resolution: Duplicate
Fix Version/s: 3.0.0
This was fixed in HIVE-18641 . Thanks for reporting.
> Cannot create table: Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
> ---
>
> Key: HIVE-18647
> URL: https://issues.apache.org/jira/browse/HIVE-18647
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Priority: Major
> Fix For: 3.0.0
>
>
> I'm using latest master branch code and mysql as metastore.
> Creating table hits this error:
> {noformat}
> 2018-02-07T22:04:55,438 ERROR [41f91bf4-bc49-4a73-baee-e2a1d79b8a4e main]
> metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of
> 10) with error: javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MTable@28d16af8" using statement
> "INSERT INTO `TBLS`
> (`TBL_ID`,`CREATE_TIME`,`CREATION_METADATA_MV_CREATION_METADATA_ID_OID`,`DB_ID`,`LAST_ACCESS_TIME`,`OWNER`,`RETENTION`,`IS_REWRITE_ENABLED`,`SD_ID`,`TBL_NAME`,`TBL_TYPE`,`VIEW_EXPANDED_TEXT`,`VIEW_ORIGINAL_TEXT`)
> VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID' in 'field list'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:729)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy36.createTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1506)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1412)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18619) Verification of temporary Micromanaged table atomicity is needed
[ https://issues.apache.org/jira/browse/HIVE-18619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355950#comment-16355950 ] Eugene Koifman commented on HIVE-18619: --- to test this in UT you can use HiveConf.HIVETESTMODEROLLBACKTXN after you set it, commit will fail - it's used simulate failures > Verification of temporary Micromanaged table atomicity is needed > - > > Key: HIVE-18619 > URL: https://issues.apache.org/jira/browse/HIVE-18619 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Minor > > Session based temporary table by HIVE-7090 had no consideration of > Micromanaged table > (MM) since there was no insert-only ACID table at its creation tije. > HIVE-18599 addressed the issue of no writes during CTTAS (Create Temporary > Table As Select) > on Micro-Managed table. But atomicity of temporary MM table is not verified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18647) Cannot create table: Unknown column 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
[
https://issues.apache.org/jira/browse/HIVE-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355949#comment-16355949
]
Vineet Garg commented on HIVE-18647:
[~jcamachorodriguez] committed HIVE-18641 which should fix this
> Cannot create table: Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
> ---
>
> Key: HIVE-18647
> URL: https://issues.apache.org/jira/browse/HIVE-18647
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Priority: Major
>
> I'm using latest master branch code and mysql as metastore.
> Creating table hits this error:
> {noformat}
> 2018-02-07T22:04:55,438 ERROR [41f91bf4-bc49-4a73-baee-e2a1d79b8a4e main]
> metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of
> 10) with error: javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MTable@28d16af8" using statement
> "INSERT INTO `TBLS`
> (`TBL_ID`,`CREATE_TIME`,`CREATION_METADATA_MV_CREATION_METADATA_ID_OID`,`DB_ID`,`LAST_ACCESS_TIME`,`OWNER`,`RETENTION`,`IS_REWRITE_ENABLED`,`SD_ID`,`TBL_NAME`,`TBL_TYPE`,`VIEW_EXPANDED_TEXT`,`VIEW_ORIGINAL_TEXT`)
> VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID' in 'field list'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:729)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy36.createTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1506)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1412)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[
https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355939#comment-16355939
]
Hive QA commented on HIVE-17063:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c50ebb3 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9075/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> insert overwrite partition onto a external table fail when drop partition
> first
> ---
>
> Key: HIVE-17063
> URL: https://issues.apache.org/jira/browse/HIVE-17063
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.2, 2.1.1, 2.2.0
>Reporter: Wang Haihua
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch,
> HIVE-17063.3.patch, HIVE-17063.4.patch
>
>
> The default value of {{hive.exec.stagingdir}} which is a relative path, and
> also drop partition on a external table will not clear the real data. As a
> result, insert overwrite partition twice will happen to fail because of the
> target data to be moved has
> already existed.
> This happened when we reproduce partition data onto a external table.
> I see the target data will not be cleared only when {{immediately generated
> data}} is child of {{the target data directory}}, so my proposal is trying
> to clear target file already existed finally whe doing rename {{immediately
> generated data}} into {{the target data directory}}
> Operation reproduced:
> {code}
> create external table insert_after_drop_partition(key string, val string)
> partitioned by (insertdate string);
> from src insert overwrite table insert_after_drop_partition partition
> (insertdate='2008-01-01') select *;
> alter table insert_after_drop_partition drop partition
> (insertdate='2008-01-01');
> from src insert overwrite table insert_after_drop_partition partition
> (insertdate='2008-01-01') select *;
> {code}
> Stack trace:
> {code}
> 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main]
> exec.Task: Failed with exception java.io.IOException: rename for src path:
[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[
https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355893#comment-16355893
]
Sahil Takiar commented on HIVE-18378:
-
[~kellyzly] yes feel free to take this up. I spent some time working on this a
few weeks ago, but hit a few issues. I attached a patch of what I had done so
far. Feel free to use that the basis of this work (or not).
The main issue I was hitting was that the {{CombineEquivalentWorkResolver}} was
being triggered in a lot of different queries. More places than I thought it
should be.
> Explain plan should show if a Map/Reduce Work is being cached
> -
>
> Key: HIVE-18378
> URL: https://issues.apache.org/jira/browse/HIVE-18378
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18378.1.patch
>
>
> It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}}
> objects are being cached by Spark.
> The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark
> cache-ing, so we should be able to modify it so that it displays if a work
> object will be cached or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[
https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355906#comment-16355906
]
Hive QA commented on HIVE-18456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909522/HIVE-18456.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 12978 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_7]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyDBName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyTableName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexNonExistingDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexNonExistingTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexNullDBName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexNullName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexNullTableName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetNonExistingIndex[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesEmptyDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesEmptyList[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesEmptyTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesInvalidDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesInvalidTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesNullDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesNullTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNamesWithDifferentNums[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexNames[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesEmptyDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesEmptyList[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesEmptyTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesInvalidDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesInvalidTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesNullDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesNullTable[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexesWithDifferentNums[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testListIndexes[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
[jira] [Comment Edited] (HIVE-18647) Cannot create table: Unknown column 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
[
https://issues.apache.org/jira/browse/HIVE-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355951#comment-16355951
]
Jesus Camacho Rodriguez edited comment on HIVE-18647 at 2/7/18 7:39 PM:
This was fixed in HIVE-18641 . Thanks for reporting. Please reopen if you still
see the issue.
was (Author: jcamachorodriguez):
This was fixed in HIVE-18641 . Thanks for reporting.
> Cannot create table: Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
> ---
>
> Key: HIVE-18647
> URL: https://issues.apache.org/jira/browse/HIVE-18647
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Priority: Major
> Fix For: 3.0.0
>
>
> I'm using latest master branch code and mysql as metastore.
> Creating table hits this error:
> {noformat}
> 2018-02-07T22:04:55,438 ERROR [41f91bf4-bc49-4a73-baee-e2a1d79b8a4e main]
> metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of
> 10) with error: javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MTable@28d16af8" using statement
> "INSERT INTO `TBLS`
> (`TBL_ID`,`CREATE_TIME`,`CREATION_METADATA_MV_CREATION_METADATA_ID_OID`,`DB_ID`,`LAST_ACCESS_TIME`,`OWNER`,`RETENTION`,`IS_REWRITE_ENABLED`,`SD_ID`,`TBL_NAME`,`TBL_TYPE`,`VIEW_EXPANDED_TEXT`,`VIEW_ORIGINAL_TEXT`)
> VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
> 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID' in 'field list'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:729)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy36.createTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1506)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1412)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Youngjin Park >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356000#comment-16356000
]
Yongzhi Chen commented on HIVE-17735:
-
I am not clear about the change related to lockForUpdate.
If you closeall for the query, will you release the lock acquired by the query
too?
> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch
>
>
> In ObjectStore.addNotificationEvent():
> {code}
> Query objectQuery = pm.newQuery(MNotificationNextId.class);
> Collection ids = (Collection)
> objectQuery.execute();
> {code}
> The query is never closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355980#comment-16355980 ] Sergey Shelukhin commented on HIVE-18636: - Some numbers appear to have changed to a different set of numbers. Looks good to me! +1 > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18586) Upgrade Derby to 10.14.1.0
[ https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18586: --- Attachment: HIVE-18586.4.patch > Upgrade Derby to 10.14.1.0 > -- > > Key: HIVE-18586 > URL: https://issues.apache.org/jira/browse/HIVE-18586 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch, > HIVE-18586.3.patch, HIVE-18586.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM
[ https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355867#comment-16355867 ] Oleksiy Sayankin commented on HIVE-18541: - Updated the patch. > Secure HS2 web UI with PAM > -- > > Key: HIVE-18541 > URL: https://issues.apache.org/jira/browse/HIVE-18541 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch > > > Secure HS2 web UI with PAM. Add two new properties > * hive.server2.webui.use.pam > * Default value: false > * Description: If true, the HiveServer2 WebUI will be secured with PAM > * hive.server2.webui.pam.authenticator > * Default value: org.apache.hive.http.security.PamAuthenticator > * Description: Class for PAM authentication -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Attachment: HIVE-18359.9.patch > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, > HIVE-18359.3.patch, HIVE-18359.4.patch, HIVE-18359.5.patch, > HIVE-18359.6.patch, HIVE-18359.7.patch, HIVE-18359.8.patch, HIVE-18359.9.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16605:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Failures are unrelated. Pushed to master
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch, HIVE-16605.5.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16605:
---
Labels: TODOC3.0 (was: )
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch, HIVE-16605.5.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Reporter: Youngjin Park (was: Sergey Shelukhin) > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Youngjin Park >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[
https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355831#comment-16355831
]
Hive QA commented on HIVE-18456:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 31207ed |
| Default Java | 1.8.0_111 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9074/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add some tests for HIVE-18367 to check that the table information contains
> the query correctly
> --
>
> Key: HIVE-18456
> URL: https://issues.apache.org/jira/browse/HIVE-18456
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18456.1.patch
>
>
> This cannot be tested with a CliDriver test so add a java test to check the
> output of 'describe extended', which is changed by HIVE-18367
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[
https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18575:
--
Comment: was deleted
(was: what was the point of renaming isAcid to isFullAcid? {{isAcid}} was used
everywhere to mean full acid table.
it now leads to code like
"compBuilder.setIsAcid(AcidUtils.isTransactionalTable(t));" which will cause
confusion?)
> ACID properties usage in jobconf is ambiguous for MM tables
> ---
>
> Key: HIVE-18575
> URL: https://issues.apache.org/jira/browse/HIVE-18575
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18575.01.patch, HIVE-18575.patch
>
>
> Vectorization checks for ACID table trigger for MM tables where they don't
> apply. Other places seem to set the setting for transactional case while most
> of the code seems to assume it implies full acid.
> Overall, many places in the code use the settings directly or set the ACID
> flag without setting the ACID properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355929#comment-16355929 ] Eugene Koifman commented on HIVE-18575: --- made some comments on RB > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.01.patch, HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Description: This removes the slowness associated with pointless metastore calls when ACID update/delete queries affect a large number of partitions. > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Youngjin Park >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18643.patch > > > This removes the slowness associated with pointless metastore calls when ACID > update/delete queries affect a large number of partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18619) Verification of temporary Micromanaged table atomicity is needed
[ https://issues.apache.org/jira/browse/HIVE-18619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355885#comment-16355885 ] Steve Yeom commented on HIVE-18619: --- A possible test case scenario for this jira is: 1. Start a Session 2. CREATE TEMPORARY TABLE mm_tab in MM(Micro-managed). 3. INSERT INTO mm_tab -- SUCCEEDED 4. INSERT INTO mm_tab -- FAILED and should leave the current statement's delta directory which should be recognized as aborted 5. SELECT FROM mm_tab -- Should only return the data at 3. This simple test case may verify basic case for atomicity of operations on TEMPORARY MM table. The patch for writeId for Replication HIVE-18192 may will affect the behavior when checked in. > Verification of temporary Micromanaged table atomicity is needed > - > > Key: HIVE-18619 > URL: https://issues.apache.org/jira/browse/HIVE-18619 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Minor > > Session based temporary table by HIVE-7090 had no consideration of > Micromanaged table > (MM) since there was no insert-only ACID table at its creation tije. > HIVE-18599 addressed the issue of no writes during CTTAS (Create Temporary > Table As Select) > on Micro-Managed table. But atomicity of temporary MM table is not verified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355909#comment-16355909 ] Prasanth Jayachandran commented on HIVE-18359: -- minor change suggested by [~kgyrtkirk] > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, > HIVE-18359.3.patch, HIVE-18359.4.patch, HIVE-18359.5.patch, > HIVE-18359.6.patch, HIVE-18359.7.patch, HIVE-18359.8.patch, HIVE-18359.9.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355910#comment-16355910
]
Vineet Garg commented on HIVE-16605:
Need to document the following:
* Hive Conf {{hive.constraint.notnull.enforce}}
* NOT NULL constraint is enforced for INSERT/MERGE/UPDATE statements.
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch, HIVE-16605.5.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18637) WorkloadManagent Event Summary leaving subscribedCounters and currentCounters fields empty
[
https://issues.apache.org/jira/browse/HIVE-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355981#comment-16355981
]
Sergey Shelukhin commented on HIVE-18637:
-
Why would it do that? It makes sense that when the query moves to a different
pool with different triggers, the list of the relevant counters changes.
Or am I missing something?
> WorkloadManagent Event Summary leaving subscribedCounters and currentCounters
> fields empty
> --
>
> Key: HIVE-18637
> URL: https://issues.apache.org/jira/browse/HIVE-18637
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18637.1.patch, HIVE-18637.2.patch
>
>
> subscribedCounters and currentCounters values are empty when trigger results
> in MOVE event
> WorkloadManager Events Summary
> {noformat}
> INFO : {
> "queryId" : "hive_20180205214449_d2955891-e3b2-4ac3-bca9-5d2a53feb8c0",
> "queryStartTime" : 1517867089060,
> "queryEndTime" : 1517867144341,
> "queryCompleted" : true,
> "queryWmEvents" : [ {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : "pool2",
> "clusterPercent" : 30.0
> },
> "eventStartTimestamp" : 1517867094797,
> "eventEndTimestamp" : 1517867094798,
> "eventType" : "GET",
> "elapsedTime" : 1
> }, {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : "pool1",
> "clusterPercent" : 70.0
> },
> "eventStartTimestamp" : 1517867139886,
> "eventEndTimestamp" : 1517867139887,
> "eventType" : "MOVE",
> "elapsedTime" : 1
> }, {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : null,
> "clusterPercent" : 0.0
> },
> "eventStartTimestamp" : 1517867144360,
> "eventEndTimestamp" : 1517867144360,
> "eventType" : "RETURN",
> "elapsedTime" : 0
> } ],
> "appliedTriggers" : [ {
> "name" : "too_large_write_triger",
> "expression" : {
> "counterLimit" : {
> "limit" : 10240,
> "name" : "HDFS_BYTES_WRITTEN"
> },
> "predicate" : "GREATER_THAN"
> },
> "action" : {
> "type" : "MOVE_TO_POOL",
> "poolName" : "pool1"
> },
> "violationMsg" : "Trigger { name: too_large_write_triger, expression:
> HDFS_BYTES_WRITTEN > 10240, action: MOVE TO pool1 } violated. Current value:
> 5096345"
> } ],
> "subscribedCounters" : [ ],
> "currentCounters" : { },
> "elapsedTime" : 55304
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: (was: HIVE-18625.2.patch)
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356278#comment-16356278
]
Hive QA commented on HIVE-18629:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909544/HIVE-18629.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12969 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9080/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909544 - PreCommit-HIVE-Build
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Attachment: HIVE-18359.11.patch > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.10.patch, > HIVE-18359.11.patch, HIVE-18359.2.patch, HIVE-18359.3.patch, > HIVE-18359.4.patch, HIVE-18359.5.patch, HIVE-18359.6.patch, > HIVE-18359.7.patch, HIVE-18359.8.patch, HIVE-18359.9.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[ https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356325#comment-16356325 ] KaiXu edited comment on HIVE-14171 at 2/8/18 1:16 AM: -- Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30) at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) ... 21 more was (Author: kaixu): Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at
[jira] [Updated] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18626: -- Attachment: HIVE-18626.3.patch > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch, > HIVE-18626.3.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356328#comment-16356328 ] Daniel Dai commented on HIVE-18626: --- Another case need to pass configuration to task. > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch, > HIVE-18626.3.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356410#comment-16356410
]
Sahil Takiar commented on HIVE-18034:
-
Actually looks like Spark's notion of totalDuration is how long a Spark task
takes to run (exposed by {{TaskInfo#duration()}}).
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356094#comment-16356094 ] Jason Dere commented on HIVE-18513: --- Rebasing patch after HIVE-17991 > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch, > HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch, HIVE-18513.6.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.2.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356139#comment-16356139
]
Hive QA commented on HIVE-18573:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 6 new + 60 unchanged - 0 fixed
= 66 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9077/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9077/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch,
> HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch,
> HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during
> Query planning. It will be more beneficial to use proper calcite operators,
> this is to prepare the ground for pushing complex expressions to
> Druid-calcite adapter.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17875) Vectorization support for complex types breaks parquet vectorization
[
https://issues.apache.org/jira/browse/HIVE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-17875:
--
Assignee: (was: Vihang Karajgaonkar)
> Vectorization support for complex types breaks parquet vectorization
>
>
> Key: HIVE-17875
> URL: https://issues.apache.org/jira/browse/HIVE-17875
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> HIVE-16589 introduced support for complex types for vectorized execution. It
> introduces two new configs {{hive.vectorized.complex.types.enabled}} and
> {{hive.vectorized.groupby.complex.types.enabled}} which default to true and
> control whether {{Vectorizer}} creates a vectorized execution plan for
> queries using complex types. Since Parquet fileformat does not support
> vectorization for complex types yet, any query running on parquet tables with
> complex types current fails with a RuntimeException complaining that the
> complex type is not supported. We should improve the logic in Vectorizer to
> check if the FileinputFormat supports complex types and if not it should not
> vectorize the query plan.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17397) vector_outer_join4.q.out explain plan not formatted correctly
[
https://issues.apache.org/jira/browse/HIVE-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-17397:
--
Assignee: (was: Vihang Karajgaonkar)
> vector_outer_join4.q.out explain plan not formatted correctly
> -
>
> Key: HIVE-17397
> URL: https://issues.apache.org/jira/browse/HIVE-17397
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Priority: Trivial
>
> {{vector_outer_join4.q}} user {{explain vectorization detail formatted}}
> which just dumps a JSON string without any indentations and new lines.
> Either, there is no option for {{explain vectorization detail formatted}} and
> it should just be {{explain vectorization detail}} or this may be a bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18649: -- Component/s: Transactions > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There also exists a JobConf field "job" in HIF; and a separate JobConf input > argument to some methods. These methods apply some changes to one jobconf and > some to another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: HIVE-18614.patch > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356232#comment-16356232
]
Hive QA commented on HIVE-18629:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 1 new + 4 unchanged -
2 fixed = 5 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed
= 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-ql.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356273#comment-16356273 ] Ashutosh Chauhan commented on HIVE-18573: - Some of test failures seems related. > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators, > this is to prepare the ground for pushing complex expressions to > Druid-calcite adapter. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356362#comment-16356362 ] Saijin Huang commented on HIVE-18578: - [~kgyrtkirk] thank you for your commit! > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Status: Patch Available (was: Open) > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356403#comment-16356403 ] Prasanth Jayachandran commented on HIVE-18653: -- [~djaiswal] can you plz take a look? > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: HIVE-18653.1.patch > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18653: > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[
https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356420#comment-16356420
]
Hive QA commented on HIVE-18636:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909556/HIVE-18636.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9082/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9082/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9082/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909556 - PreCommit-HIVE-Build
> fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
> --
>
> Key: HIVE-18636
> URL: https://issues.apache.org/jira/browse/HIVE-18636
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18636.01.patch
>
>
> need to update expected result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356423#comment-16356423
]
Prasanth Jayachandran commented on HIVE-18547:
--
I cannot repro this issue locally since this is a race condition. This patch
updates the cluster fraction of previously recorded events when the cluster
fraction changes.
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Status: Patch Available (was: Open)
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Affects Version/s: 3.0.0
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356235#comment-16356235
]
Sergey Shelukhin commented on HIVE-18629:
-
Updated to also handle isRepeating case.
[~mmccline] I don't think the changes around noNulls are necessary... they can
only become relevant if input vector is invalid in the first place and has
isNull elements when noNulls is set.
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18586) Upgrade Derby to 10.14.1.0
[ https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356274#comment-16356274 ] Aihua Xu commented on HIVE-18586: - [~alangates] Can you help take a look at the test change [~janulatha] is making? What is the original purpose to add SecurityManager to check System.exit()? Thanks. > Upgrade Derby to 10.14.1.0 > -- > > Key: HIVE-18586 > URL: https://issues.apache.org/jira/browse/HIVE-18586 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch, > HIVE-18586.3.patch, HIVE-18586.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17837:
Attachment: HIVE-17837.2.patch
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17837.1.patch, HIVE-17837.2.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18598) Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356299#comment-16356299 ] Ashutosh Chauhan commented on HIVE-18598: - +1 > Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch, HIVE-18598.2.patch, > HIVE-18598.3.patch > > > HIVE-16605 is enabling/enforcing NOT NULL constraint. But since Hive do not > manage the data for external tables and can not enforce constraints it > doesn't make sense to allow NOT NULL constraints to be enabled/enforced on > external table. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356325#comment-16356325
]
KaiXu commented on HIVE-14171:
--
Found similar issue with hive.vectorized.use.row.serde.deserialize=true on
TPC-DS query12, parquet file format:
Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most
recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2):
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228)
... 17 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 21 more
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
>
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356365#comment-16356365
]
liyunzhang commented on HIVE-18340:
---
[~stakiar]: {quote}
Hive-on-Tez's has an implementation of DynamicValueRegistry that uses some
special Tez APIs such as ProcessorContext#waitForAllInputsReady, how are we
simulating this in HoS?
{quote}
ProcessorContext#waitForAllInputsReady is called by
{{org.apache.hadoop.hive.ql.exec.tez.DynamicValueRegistryTez#init}} to read the
runtime filter info. For HoS, I guess [~Jk_self] will read the info from hdfs
which is similar as Spark DPP.
If my understanding is not right, [~stakiar], [~Jk_Self] please tell me.
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
>
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Ke Jia
>Assignee: Ke Jia
>Priority: Major
> Attachments: HIVE-18340.1.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Attachment: HIVE-18547.1.patch
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18547:
Assignee: Prasanth Jayachandran
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[
https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356295#comment-16356295
]
Hive QA commented on HIVE-15911:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 429 unchanged - 0
fixed = 431 total (was 429) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create a view based on another view throws an exception “FAILED:
> NullPointerException null”
> ---
>
> Key: HIVE-15911
> URL: https://issues.apache.org/jira/browse/HIVE-15911
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.1.1
> Environment: hive2.1.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: 0001-Hive-15911.patch
>
>
> when I create a new view based on another view I an exception “FAILED:
> NullPointerException null”:
> hive> create view view2(a,b) as select a, b from view1; //view1 is another
> view
> FAILED: NullPointerException null
> hive>
> The hive log show error stack：
> 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116))
> - FAILED: NullPointerException null
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459)
> at
[jira] [Commented] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[
https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356353#comment-16356353
]
Hive QA commented on HIVE-15911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853014/0001-Hive-15911.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_as_select_view]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9081/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9081/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9081/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853014 - PreCommit-HIVE-Build
> Create a view based on another view throws an exception “FAILED:
> NullPointerException null”
> ---
>
> Key: HIVE-15911
> URL: https://issues.apache.org/jira/browse/HIVE-15911
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.1.1
> Environment: hive2.1.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: 0001-Hive-15911.patch
>
>
> when I create a new view based on another view I an exception “FAILED:
> NullPointerException null”:
> hive> create view view2(a,b) as select a, b from view1; //view1 is another
> view
> FAILED: NullPointerException null
> hive>
> The hive log show error stack：
> 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116))
> - FAILED: NullPointerException null
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1239)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
>
[jira] [Commented] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[
https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356369#comment-16356369
]
Hive QA commented on HIVE-18636:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 18 new + 261 unchanged - 18
fixed = 279 total (was 279) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9082/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9082/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
> --
>
> Key: HIVE-18636
> URL: https://issues.apache.org/jira/browse/HIVE-18636
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18636.01.patch
>
>
> need to update expected result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356393#comment-16356393
]
Sahil Takiar edited comment on HIVE-18034 at 2/8/18 2:33 AM:
-
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime + executorDeserializeTime}} should give
us the total executor duration, and {{jvmGCTime}} should give the GC time. I
think a {{LOG#warn}} and a warning on the console should be printed.
was (Author: stakiar):
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime}} + {{ executorDeserializeTime}} should
give us the total executor duration, and {{jvmGCTime}} should give the GC time.
I think a {{LOG#warn}} and a warning on the console should be printed.
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356393#comment-16356393
]
Sahil Takiar commented on HIVE-18034:
-
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime}} + {{ executorDeserializeTime}} should
give us the total executor duration, and {{jvmGCTime}} should give the GC time.
I think a {{LOG#warn}} and a warning on the console should be printed.
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356392#comment-16356392
]
Vihang Karajgaonkar commented on HIVE-14171:
AFAIK {{hive.vectorized.use.row.serde.deserialize}} is not supported for
Parquet. Does it fail if you set to false as well? Many fixes for Parquet
vectorized reader went in 2.3.0. So it would be good to check using 2.3.0 if
its possible.
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356394#comment-16356394
]
Vihang Karajgaonkar commented on HIVE-14171:
cc [~Ferd]
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18654) Add Hiveserver2 specific HADOOP_OPTS environment variable
[ https://issues.apache.org/jira/browse/HIVE-18654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-18654: -- > Add Hiveserver2 specific HADOOP_OPTS environment variable > -- > > Key: HIVE-18654 > URL: https://issues.apache.org/jira/browse/HIVE-18654 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > HIVE-2665 added support to include metastore specific HADOOP_OPTS variable. > This is helpful in debugging especially if you want to add some jvm > parameters to metastore's process. A similar setting for Hiveserver2 is > missing and could be very helpful in debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: HIVE-18653.1.patch > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: (was: HIVE-18653.1.patch) > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[ https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-18340: -- Attachment: HIVE-18340.2.patch > Dynamic Min-Max/BloomFilter runtime-filtering in HoS > > > Key: HIVE-18340 > URL: https://issues.apache.org/jira/browse/HIVE-18340 > Project: Hive > Issue Type: New Feature > Components: Spark >Affects Versions: 3.0.0 >Reporter: Ke Jia >Assignee: Ke Jia >Priority: Major > Attachments: HIVE-18340.1.patch, HIVE-18340.2.patch > > > Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269 > and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18629:
Attachment: HIVE-18629.02.patch
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356322#comment-16356322
]
Sahil Takiar commented on HIVE-17837:
-
Rebased patch. [~lirui] could you take a look? I checked the code and there
shouldn't be much additional overhead to calling
{{RemoteSparkJobStatus#isRemoteActive}} multiple times. The method essentially
boils down to checking a few boolean variables, and acquiring / releasing a
lock while checking the state of the underlying {{SocketChannel}}.
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17837.1.patch, HIVE-17837.2.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[ https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356325#comment-16356325 ] KaiXu edited comment on HIVE-14171 at 2/8/18 1:17 AM: -- Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format. environment: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 stack trace: Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30) at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) ... 21 more was (Author: kaixu): Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at
[jira] [Commented] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356343#comment-16356343 ] Ashutosh Chauhan commented on HIVE-18607: - +1 > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356426#comment-16356426
]
Ke Jia commented on HIVE-18340:
---
[~stakiar]：
{quote}Hive-on-Tez's has an implementation of DynamicValueRegistry that uses
some special Tez APIs such as ProcessorContext#waitForAllInputsReady, how are
we simulating this in HoS?
{quote}
[~kellyzly],Yes, For HoS, I flush the runtime filter info (min/max and bloom
filter) to hdfs in SparkRuntimeFilterPruningSinkOperator operator and get the
info from hdfs in SparkRuntimeFilterPruner , which is similar as
SparkPartitionPruningSinkOperator and SparkDynamicPartitionPruner class in
Spark DPP.
{quote}It would be nice to have some qtests to help visualize what the explain
plan with RF would look like
{quote}
I upload the HIVE-18340.2.patch to add qtest "spark_runtime_filter_pruning.q"
and "spark_runtime_filter_pruning.q.out".
Thanks [~stakiar], [~kellyzly] for your review!
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
>
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Ke Jia
>Assignee: Ke Jia
>Priority: Major
> Attachments: HIVE-18340.1.patch, HIVE-18340.2.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
[
https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356127#comment-16356127
]
Ashutosh Chauhan commented on HIVE-18595:
-
Some of udf test failures look related.
> UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
>
>
> Key: HIVE-18595
> URL: https://issues.apache.org/jira/browse/HIVE-18595
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18595.patch, HIVE-18595.patch
>
>
> {code}
> 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main]
> ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong
> arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only
> string/date/timestamp types
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:456 Wrong arguments
> ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only
> string/date/timestamp types
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1394)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:235)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:181)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11847)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11780)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genGBLogicalPlan(CalcitePlanner.java:3140)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:4330)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1407)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1354)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1159)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1175)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:422)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11393)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:639)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1632)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1382)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:240)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1331)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1305)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
> at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Updated] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18513: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) I've committed this to master. Thanks for review [~jcamachorodriguez] > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch, > HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch, HIVE-18513.6.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356212#comment-16356212 ] Sergey Shelukhin commented on HIVE-18607: - [~ashutoshc] can you take a look? not sure if anyone is familiar with HBase handler anymore. > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18607: Attachment: HIVE-18607.04.patch > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356040#comment-16356040
]
Sahil Takiar commented on HIVE-17835:
-
[~csun] noticed you made some changes to this logic recently, could you take a
look at the RB: https://reviews.apache.org/r/6/
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17835.1.patch, HIVE-17835.2.patch,
> HIVE-17835.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
>
[jira] [Commented] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
[
https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356051#comment-16356051
]
Hive QA commented on HIVE-18595:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 6 new + 27 unchanged - 2 fixed
= 33 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / cf3394d |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9076/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9076/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
>
>
> Key: HIVE-18595
> URL: https://issues.apache.org/jira/browse/HIVE-18595
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18595.patch, HIVE-18595.patch
>
>
> {code}
> 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main]
> ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong
> arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only
> string/date/timestamp types
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:456 Wrong arguments
> ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only
> string/date/timestamp types
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1394)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:235)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:181)
> at
>
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356047#comment-16356047
]
Sahil Takiar commented on HIVE-18340:
-
Some high level questions:
* Hive-on-Tez's has an implementation of {{DynamicValueRegistry}} that uses
some special Tez APIs such as {{ProcessorContext#waitForAllInputsReady}}, how
are we simulating this in HoS?
* It would be nice to have some qtests to help visualize what the explain plan
with RF would look like
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
>
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Ke Jia
>Assignee: Ke Jia
>Priority: Major
> Attachments: HIVE-18340.1.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18642: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[ https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356164#comment-16356164 ] Thejas M Nair commented on HIVE-17983: -- +1 pending tests Thanks for adding the docker support for being able to test scripts changes easily! That really helps with developer productivity! It would be good to get this in soon, as there are some other db script changes in pipeline blocked on this (ie, they would conflict with this) > Make the standalone metastore generate tarballs etc. > > > Key: HIVE-17983 > URL: https://issues.apache.org/jira/browse/HIVE-17983 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17983.2.patch, HIVE-17983.3.patch, > HIVE-17983.4.patch, HIVE-17983.patch > > > In order to be separately installable the standalone metastore needs its own > tarballs, startup scripts, etc. All of the SQL installation and upgrade > scripts also need to move from metastore to standalone-metastore. > I also plan to create Dockerfiles for different database types so that > developers can test the SQL installation and upgrade scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18637) WorkloadManagent Event Summary leaving subscribedCounters and currentCounters fields empty
[
https://issues.apache.org/jira/browse/HIVE-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356070#comment-16356070
]
Sergey Shelukhin commented on HIVE-18637:
-
k makes sense, +1 pending tests
> WorkloadManagent Event Summary leaving subscribedCounters and currentCounters
> fields empty
> --
>
> Key: HIVE-18637
> URL: https://issues.apache.org/jira/browse/HIVE-18637
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18637.1.patch, HIVE-18637.2.patch
>
>
> subscribedCounters and currentCounters values are empty when trigger results
> in MOVE event
> WorkloadManager Events Summary
> {noformat}
> INFO : {
> "queryId" : "hive_20180205214449_d2955891-e3b2-4ac3-bca9-5d2a53feb8c0",
> "queryStartTime" : 1517867089060,
> "queryEndTime" : 1517867144341,
> "queryCompleted" : true,
> "queryWmEvents" : [ {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : "pool2",
> "clusterPercent" : 30.0
> },
> "eventStartTimestamp" : 1517867094797,
> "eventEndTimestamp" : 1517867094798,
> "eventType" : "GET",
> "elapsedTime" : 1
> }, {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : "pool1",
> "clusterPercent" : 70.0
> },
> "eventStartTimestamp" : 1517867139886,
> "eventEndTimestamp" : 1517867139887,
> "eventType" : "MOVE",
> "elapsedTime" : 1
> }, {
> "wmTezSessionInfo" : {
> "sessionId" : "157866e5-ed1c-4abd-9846-db76b91c1124",
> "poolName" : null,
> "clusterPercent" : 0.0
> },
> "eventStartTimestamp" : 1517867144360,
> "eventEndTimestamp" : 1517867144360,
> "eventType" : "RETURN",
> "elapsedTime" : 0
> } ],
> "appliedTriggers" : [ {
> "name" : "too_large_write_triger",
> "expression" : {
> "counterLimit" : {
> "limit" : 10240,
> "name" : "HDFS_BYTES_WRITTEN"
> },
> "predicate" : "GREATER_THAN"
> },
> "action" : {
> "type" : "MOVE_TO_POOL",
> "poolName" : "pool1"
> },
> "violationMsg" : "Trigger { name: too_large_write_triger, expression:
> HDFS_BYTES_WRITTEN > 10240, action: MOVE TO pool1 } violated. Current value:
> 5096345"
> } ],
> "subscribedCounters" : [ ],
> "currentCounters" : { },
> "elapsedTime" : 55304
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18639) HiveMetaStoreFsImpl#deleteDir should check return value of FileUtils#moveToTrash
[
https://issues.apache.org/jira/browse/HIVE-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18639:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Vihang for the review.
> HiveMetaStoreFsImpl#deleteDir should check return value of
> FileUtils#moveToTrash
>
>
> Key: HIVE-18639
> URL: https://issues.apache.org/jira/browse/HIVE-18639
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18639.1.patch
>
>
> {{HiveMetaStoreFsImpl#deleteDir}} moves a dir to a the trash, but then
> doesn't check if {{FileUtils#moveToTrash}} returns true or not. Then it
> checks if the file exists, and throes an exception if it doesn't. If
> {{FileUtils#moveToTrash}} returns {{true}} then the {{exists}} check is not
> necessary. Before HIVE-15880 the code was reading the return value of
> {{delete}} and then conditionally ran the {{exists}} check, we should go back
> to the old behavior. This can also cause problems on filesystems like S3.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356105#comment-16356105
]
Andrew Sherman commented on HIVE-18625:
---
{color:#33}Test failures appear unrelated.{color}
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356149#comment-16356149 ] Sergey Shelukhin commented on HIVE-18649: - cc [~ekoifman] > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There is also a job field with JobConf, and an input to some methods that is > also JobConf; some methods apply some changes to one jobconf and some to > another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Attachment: HIVE-17835.3.patch
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17835.1.patch, HIVE-17835.2.patch,
> HIVE-17835.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Status: Patch Available (was: Open)
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17835.1.patch, HIVE-17835.2.patch,
> HIVE-17835.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
>
[jira] [Updated] (HIVE-17835) HS2 Logs print unnecessary stack trace when HoS query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17835:
Status: Open (was: Patch Available)
> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17835
> URL: https://issues.apache.org/jira/browse/HIVE-17835
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17835.1.patch, HIVE-17835.2.patch,
> HIVE-17835.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
>
[jira] [Commented] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356106#comment-16356106
]
Yongzhi Chen commented on HIVE-17735:
-
By theory, it is. Just to double check, for our pattern to close query is after
the transaction committed. And it is not difficult to hold the for update query
until end of transaction to close.
> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch
>
>
> In ObjectStore.addNotificationEvent():
> {code}
> Query objectQuery = pm.newQuery(MNotificationNextId.class);
> Collection ids = (Collection)
> objectQuery.execute();
> {code}
> The query is never closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
[
https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356117#comment-16356117
]
Hive QA commented on HIVE-18595:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909532/HIVE-18595.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 12987 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_udf] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_udf]
(batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin6]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_udf]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[timestamp_udf]
(batchId=142)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNoDbName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFDateDiff.testTimestampToDate
(batchId=261)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=231)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9076/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9076/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9076/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909532 - PreCommit-HIVE-Build
> UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
>
>
> Key: HIVE-18595
> URL: https://issues.apache.org/jira/browse/HIVE-18595
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18595.patch, HIVE-18595.patch
>
>
> {code}
> 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main]
> ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong
> arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only
> string/date/timestamp types
>
