Re: Review Request 65002: HIVE-18229 add the unmanaged mapping command

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65002/#review194924
---




ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 4057 (original), 4062 (patched)


hm, will take a look at this next week


- Sergey Shelukhin


On Jan. 6, 2018, 3:25 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65002/
> ---
> 
> (Updated Jan. 6, 2018, 3:25 a.m.)
> 
> 
> Review request for hive and Harish Jaiprakash.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 59bef52b74 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> c413bc5126 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 1b051f2e58 
>   ql/src/test/queries/clientpositive/resourceplan.q ce684ed51b 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 29c01a8b43 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  5bfa62485b 
> 
> 
> Diff: https://reviews.apache.org/r/65002/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 65002: HIVE-18229 add the unmanaged mapping command

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65002/
---

Review request for hive and Harish Jaiprakash.


Repository: hive-git


Description
---

see jira


Diffs
-

  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 59bef52b74 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
c413bc5126 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 1b051f2e58 
  ql/src/test/queries/clientpositive/resourceplan.q ce684ed51b 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 29c01a8b43 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 5bfa62485b 


Diff: https://reviews.apache.org/r/65002/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 64490: HIVE-14498

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64490/
---

(Updated Jan. 6, 2018, 3:04 a.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Bugs: HIVE-14498
https://issues.apache.org/jira/browse/HIVE-14498


Repository: hive-git


Description
---

HIVE-14498


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
1dc7501291691c333ddda741b8f0f461a69a9a65 
  data/files/ssb/customer/0_0 PRE-CREATION 
  data/files/ssb/date/0_0 PRE-CREATION 
  data/files/ssb/lineorder/0_0 PRE-CREATION 
  data/files/ssb/part/0_0 PRE-CREATION 
  data/files/ssb/supplier/0_0 PRE-CREATION 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 eded92e1c16bd43bb665fd608c5262767c14ef8e 
  itests/hive-unit/src/main/java/org/hadoop/hive/jdbc/SSLTestUtils.java 
6cbcf8ca7cdea8b99736b50b38634c077895f5d8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
 260c9473748b330399963ff25956e1b576793e33 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
 2bb06d582c0ae2902ab29d7ab8d9e25788389698 
  metastore/scripts/upgrade/derby/047-HIVE-14498.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 
ea75082539d57c1b640a6bfa2bf072dfbd72999e 
  metastore/scripts/upgrade/derby/hive-txn-schema-3.0.0.derby.sql 
52713df30c66b34a3d5b815ca2814e7dca1e45a6 
  metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 
1f2647dfbf3263feda9afd98ab6767d7ea3d0557 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 
59bef52b74723fc6ceafc10b7f62abb16c24cfec 
  metastore/scripts/upgrade/mssql/032-HIVE-14498.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 
402dfce4dbf7498b14b501cdc813cbf65a4a2153 
  metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 
864a5e5bd5c06810cab9d2f09d3b968845059a7a 
  metastore/scripts/upgrade/mysql/047-HIVE-14498.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 
f7312f037249dd64f5ef6207b2e50a5dea3e3f58 
  metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql 
1df32c4b3548d385e8861b9312042cc25bdf84d7 
  metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql 
caa059d893635e6afadab867760cb035b3a111d4 
  metastore/scripts/upgrade/oracle/047-HIVE-14498.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 
1cfe768ff928af33b1fe61c9717df6c647735038 
  metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql 
12c24a5863e0b8382c643f0404700e0243585db0 
  metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
33174c8a9a9bc768919ddda406324eda05aaf313 
  metastore/scripts/upgrade/postgres/046-HIVE-14498.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
2c2d3a14ded8c575c72df628f1d63480bf1ce4de 
  metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql 
1fa99aff5fcbbb96ed51b9c02850dfabcd6d3d76 
  metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
01d359e5f4c632793ac8e3fb67aa6ea5492dac54 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
6d48783d48581fb96ea1b5ded23ce0d549dc80a6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
cf8386bac557526c162552a0abb068aa0e50469f 
  
ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryUpdateHook.java
 a57e4c888b204388b393bb173e2eac91c867137a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
20d7593d8867e1dfcd43dff99c535119547c238f 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 2b1023a54d163cf0b084916c333cbfe17d6f6a0d 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
0debff669a5d64d87bedd1c11a856cf561e46590 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
28b4cfeaf77a980899bcfa72d6b74abb5704ec42 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
c413bc5126bc2e10d72021c78ca088f3fd221d4a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
5a88a96d72a83e4ce93430f47303ca4e9ed4a096 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
09aa82f1f0c1a90b08669b91615f26fb1f7cd649 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
33e30bf10fc048fdbe2bd4a78e5d6d94bd7b04d1 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
69b076a08a70fcea4f262ccbf9e063733ddd25f2 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
bbd285d54aeae3073045b2cbbb5ac4c3f6cb6b2b 
  
ql/src/test/queries/clientnegative/materialized_view_no_transactional_rewrite.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_transactional_rewrite_2.q
 PRE-CREATION 
  ql/src/test/queri

[GitHub] hive pull request #287: HIVE-17580 : Remove standalone-metastore's dependenc...

[vihangk1]

GitHub user vihangk1 opened a pull request:

https://github.com/apache/hive/pull/287

HIVE-17580 : Remove standalone-metastore's dependency with serdes

Removing the dependency on serdes for the metastore requires a series of 
changes. I have created multiple commits which hopefully would be easier to 
review. Each major commit has a descriptive commit message to give a high level 
idea of what the change is doing. There are still some bits which need to be 
completed but it would be good to a review.

Overview of all the changes done:
1. Creates a new module called serde-api under storage-api like discussed. 
Although I think we can keep it separate as well.
2. Moved List, Map, Struct, Constant, Primitive, Union ObjectInspectors to 
serde-api
3. Moved PrimitiveTypeInfo, PrimitiveTypeEntry and TypeInfo to serde-api.
4. Moved TypeInfoParser, TypeInfoFactory to serde-api
5. Added a new class which reading avro storage schema by copying the code 
from AvroSerde and AvroSerdeUtils. The parsing is done such that String value 
is first converted into TypeInfos and then into FieldSchemas bypassing the need 
for ObjectInspectors. In theory we could get rid of TypeInfos as well but that 
path was getting too difficult with lot of duplicate code between Hive and 
metastore.
6. Introduces a default storage schema reader. I noticed that most of the 
serdes use the same logic to parse the metadata information. This code should 
be refactored to a common place instead of having many copies (one in 
standalone hms and another set in multiple serdes)
7. Moved HiveChar, HiveVarchar, HiveCharWritable, HiveVarcharWritable to 
storage-api. I noticed that HiveDecimal is already in storage-api. It probably 
makes sense to move the other primitive types (timestamp, interval etc)to 
storage-api as well but it requires storage-api to be upgraded to Java 8.
8. Adds a basic test for the schema reader. I plan to add more tests as 
this code is reviewed.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #287


commit bbfb7dc44904db74a840167c02b07f50a6010b69
Author: Vihang Karajgaonkar 
Date:   2017-11-09T00:52:39Z

HIVE-17580 : Remove dependency of get_fields_with_environment_context API 
to serde

commit d54879845eff10c19bc17bda9e09dda16f6fa295
Author: Vihang Karajgaonkar 
Date:   2017-11-29T00:54:23Z

Moved List, Map, Struct OI to storage-api

commit a12d6c7ba3de598c6b6f75da1bd4efcac43036b1
Author: Vihang Karajgaonkar 
Date:   2017-11-29T04:19:39Z

Moved ConstantObjectInspector PrimitiveObjectInspector and 
UnionObjectInspector

commit 13fb832fc2d51958e75d5e609f6781f87449aed8
Author: Vihang Karajgaonkar 
Date:   2017-12-28T01:25:59Z

Moved PrimitiveTypeInfo to serde-api

In order to move PrimitiveTypeInfo we need to move the PrimitiveTypeEntry 
as well.
PrimitiveTypeEntry depends on PrimitiveObjectInspectorUtils which cannot be 
pulled
into serde-api. Hence the static final maps are moved to PrimitiveEntry and 
we provide
static access methods to these maps along with the register method to add 
the key
value pairs in the maps

commit 9cbc789fd3f4ced7ce66a7313c451b75a154976f
Author: Vihang Karajgaonkar 
Date:   2017-12-28T20:51:39Z

Moved the other TypeInfos to serde-api

In order to move the other TypeInfo classes to serde-api we need to move 
the serdeConstants.java
as well. This is a thrift generated class. This commit copies the 
serde.thrift instead of moving.
The only reason I did not move it is in case of backwards compatibility 
reasons (in case someone is using the thrift file location to do something).
If it is okay to move serde.thrift from serde module to serde-api we can 
delete it from serde module in a separate
change.

The other concern is there are some TypeInfo classes which do some 
validation like VarCharTypeInfo, DecimalTypeInfo.
The validating methods use the actual type implementation like HiveChar, 
HiveDecimal etc to ensure that the params
are under the correct limits. This creates a problem since we cannot bring 
in the type implementations as well to
serde-api. Currently, I have marked these as TODO and commented them out.

commit 23aa899a90d17648e560d39602a3bd29bf53661e
Author: Vihang Karajgaonkar 
Date:   2017-12-29T22:14:06Z

Moved TypeInfoParser and TypeInfoFactory to serde-api

In order to use TypeInfoParser in standalone metastore, we should move it 
to serde-api
There is a problem in moving the TypeInfoParser to serde-api which is there 
are some validation util
methods which validated the m

Re: Review Request 64999: HIVE-18004 investigate deriving app name from JDBC connection for pool mapping

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64999/#review194918
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java
Lines 37 (patched)


the comment needs to be removed


- Sergey Shelukhin


On Jan. 5, 2018, 11:51 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64999/
> ---
> 
> (Updated Jan. 5, 2018, 11:51 p.m.)
> 
> 
> Review request for hive and Harish Jaiprakash.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 45acf13201 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java bb13682a9a 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 9ff733d1bb 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql ea75082539 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 3fae382dca 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 402dfce4db 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql ed8b302caa 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql f7312f0372 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 1b0ea94435 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 1cfe768ff9 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql 99a1a25e87 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 2c2d3a14de 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 38f80fd882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 5919f3f805 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 01354af366 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 1b051f2e58 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> e90e227a96 
>   ql/src/test/queries/clientpositive/resourceplan.q a4a9b7c811 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 94cf877d0c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 9b13ea7be8 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 98125d3c3e 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java 8c993a5d85 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> bd4d90d5d1 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7fbcd13b63 
>   
> service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
>  71e53b7cc6 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> eef7a253d4 
>   
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
>  d43f1254c5 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMMapping.java
>  ec0ac3f148 
>   standalone-metastore/src/main/resources/package.jdo e3ae67d58e 
> 
> 
> Diff: https://reviews.apache.org/r/64999/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 64999: HIVE-18004 investigate deriving app name from JDBC connection for pool mapping

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64999/
---

Review request for hive and Harish Jaiprakash.


Repository: hive-git


Description
---

see jira


Diffs
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 45acf13201 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java bb13682a9a 
  metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 9ff733d1bb 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql ea75082539 
  metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 3fae382dca 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 402dfce4db 
  metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql ed8b302caa 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql f7312f0372 
  metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 1b0ea94435 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 1cfe768ff9 
  metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql 99a1a25e87 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 2c2d3a14de 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 38f80fd882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
5919f3f805 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 01354af366 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 1b051f2e58 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
e90e227a96 
  ql/src/test/queries/clientpositive/resourceplan.q a4a9b7c811 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 94cf877d0c 
  service/src/java/org/apache/hive/service/cli/CLIService.java 9b13ea7be8 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
98125d3c3e 
  service/src/java/org/apache/hive/service/cli/ICLIService.java 8c993a5d85 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
bd4d90d5d1 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
7fbcd13b63 
  
service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
 71e53b7cc6 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
eef7a253d4 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
d43f1254c5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMMapping.java
 ec0ac3f148 
  standalone-metastore/src/main/resources/package.jdo e3ae67d58e 


Diff: https://reviews.apache.org/r/64999/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18389) Print out Spark Web UI URL to the console log

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18389:
---

 Summary: Print out Spark Web UI URL to the console log
 Key: HIVE-18389
 URL: https://issues.apache.org/jira/browse/HIVE-18389
 Project: Hive
  Issue Type: Sub-task
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Should be accessible via {{SparkContext#uiWebUrl}}. It just needs to be sent 
from the {{RemoteDriver}} to HS2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[Announce] New committer: Deepak Jaiswal

[Ashutosh Chauhan]

The Project Management Committee (PMC) for Apache Hive has invited Deepak
Jaiswal to become a committer and we are pleased to announce that he has
 accepted.

Welcome, Deepak!

Thanks,
 Ashutosh

Re: Review Request 64903: HIVE-18269 LLAP: Fast llap io with slow processing pipeline can lead to OOM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64903/
---

(Updated Jan. 5, 2018, 9:59 p.m.)


Review request for hive, Gopal V and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1dc7501291 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 4dc107a6cb 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
77c8ade5c9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 52a9c238ac 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 1f3f4d2b4f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 373af76cf6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 29304977a9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 21f90a7853 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 4b0a1ceca4 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Consumer.java 43c56472f3 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
29cef30d5f 


Diff: https://reviews.apache.org/r/64903/diff/2/

Changes: https://reviews.apache.org/r/64903/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 64925: HIVE-18349: Misc metastore changes for debuggability

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64925/
---

(Updated Jan. 5, 2018, 9:44 p.m.)


Review request for hive and Thejas Nair.


Changes
---

stats related metastore events did not pass the success state correctly that 
was causing these test failures.


Bugs: HIVE-18349
https://issues.apache.org/jira/browse/HIVE-18349


Repository: hive-git


Description
---

HIVE-18349: Misc metastore changes for debuggability


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java
 1ca18b9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 f1b58c5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionContext.java
 998531f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 2e43dc8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 4ff2bb7 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 6ffc24a 


Diff: https://reviews.apache.org/r/64925/diff/7/

Changes: https://reviews.apache.org/r/64925/diff/6-7/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 64787: HIVE-18326 LLAP Tez scheduler - only preempt tasks if there's a dependency between them

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64787/
---

(Updated Jan. 5, 2018, 9:32 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1dc7501291 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 e97a267843 


Diff: https://reviews.apache.org/r/64787/diff/3/

Changes: https://reviews.apache.org/r/64787/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 64903: HIVE-18269 LLAP: Fast llap io with slow processing pipeline can lead to OOM

[Jason Dere]

> On Jan. 5, 2018, 1:32 a.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Lines 520 (patched)
> > 
> >
> > Maybe interesting to save/log the number of iterations spent in this 
> > loop, if iterations > 1?
> 
> Sergey Shelukhin wrote:
> Hmm... it's just wait time. Not sure if number of iterations matter.

The iterations would really just be a rough approximation of the time the 
reader is blocked waiting for the consumer, if that is an interesting number to 
keep track of.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64903/#review194821
---


On Jan. 3, 2018, 2:32 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64903/
> ---
> 
> (Updated Jan. 3, 2018, 2:32 a.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be83489cb3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  4dc107a6cb 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 77c8ade5c9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  52a9c238ac 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
>  1f3f4d2b4f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  373af76cf6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  29304977a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  21f90a7853 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  4b0a1ceca4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Consumer.java 
> 43c56472f3 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 29cef30d5f 
> 
> 
> Diff: https://reviews.apache.org/r/64903/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 64903: HIVE-18269 LLAP: Fast llap io with slow processing pipeline can lead to OOM

[Sergey Shelukhin]

> On Jan. 5, 2018, 1:32 a.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Lines 520 (patched)
> > 
> >
> > Maybe interesting to save/log the number of iterations spent in this 
> > loop, if iterations > 1?

Hmm... it's just wait time. Not sure if number of iterations matter.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64903/#review194821
---


On Jan. 3, 2018, 2:32 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64903/
> ---
> 
> (Updated Jan. 3, 2018, 2:32 a.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be83489cb3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  4dc107a6cb 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 77c8ade5c9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  52a9c238ac 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
>  1f3f4d2b4f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  373af76cf6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  29304977a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  21f90a7853 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  4b0a1ceca4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Consumer.java 
> 43c56472f3 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 29cef30d5f 
> 
> 
> Diff: https://reviews.apache.org/r/64903/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 64903: HIVE-18269 LLAP: Fast llap io with slow processing pipeline can lead to OOM

[Sergey Shelukhin]

> On Jan. 5, 2018, 1:07 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Lines 155 (patched)
> > 
> >
> > OrcInputFormat does some adjustments for ACID. 
> > 
> > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L1648
> > 
> > Maybe the same applies here.

It's actually unrelated. I remember there was a bug where includes would be 
null in extremely rare cases so this code was added.
However over time it seems to have been moved after columnIds assignment so it 
would still break stuff.
I am moving it back in front so the other code won't NPE.
Looks like the below code would handle ACID.


> On Jan. 5, 2018, 1:07 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Lines 419 (patched)
> > 
> >
> > log the chosen queue size for debugging. Also will be good if current 
> > queue size can be printed periodically or when limits are reached once or 
> > expose as metrics.

Limits are per fragment and never change atm. Added the logging at init time


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64903/#review194816
---


On Jan. 3, 2018, 2:32 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64903/
> ---
> 
> (Updated Jan. 3, 2018, 2:32 a.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be83489cb3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  4dc107a6cb 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 77c8ade5c9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  52a9c238ac 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
>  1f3f4d2b4f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  373af76cf6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  29304977a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  21f90a7853 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  4b0a1ceca4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Consumer.java 
> 43c56472f3 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 29cef30d5f 
> 
> 
> Diff: https://reviews.apache.org/r/64903/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18388) LLAP RecordReader may fail for ACID scan if included cols is null

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18388:
---

 Summary: LLAP RecordReader may fail for ACID scan if included cols 
is null
 Key: HIVE-18388
 URL: https://issues.apache.org/jira/browse/HIVE-18388
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


We haven't ever seen this, but it may theoretically happen. See HIVE-18269.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18387) Minimize time that REBUILD locks the materialized view

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18387:
--

 Summary: Minimize time that REBUILD locks the materialized view
 Key: HIVE-18387
 URL: https://issues.apache.org/jira/browse/HIVE-18387
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Currently, REBUILD will block the materialized view while the final move task 
is being executed. The idea for this improvement is to create the new 
materialization in a new folder (new version) and then just flip the pointer to 
the folder in the MV definition in the metastore. REBUILD operations for a 
given MV should get an exclusive lock though, i.e., they cannot be executed 
concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18386:
--

 Summary: Create dummy materialized views registry and make it 
configurable
 Key: HIVE-18386
 URL: https://issues.apache.org/jira/browse/HIVE-18386
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
have quick access when queries are planned. For debugging purposes, we will 
create a dummy materialized views registry that forwards all calls to metastore 
and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-18385:
-

 Summary: mergejoin fails with java.lang.IllegalStateException
 Key: HIVE-18385
 URL: https://issues.apache.org/jira/browse/HIVE-18385
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


mergejoin test fails with java.lang.IllegalStateException when run in 
MiniLlapLocal.

This is the query for which it fails,
[ERROR]   TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed 
with error code = 2 running "
select count(*) from tab a join tab_part b on a.key = b.key join src1 c on 
a.value = c.value" fname=mergejoin.q 


This is the stack trace,

failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, 
vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: ORC 
split generation failed with exception: java.lang.IllegalStateException: Failed 
to retrieve dynamic value for RS_12_a_key_min
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.IllegalStateException: Failed to retrieve dynamic value for 
RS_12_a_key_min
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778)
... 17 more
Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value 
for RS_12_a_key_min
at 
org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142)
at 
org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97)
at 
org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93)
at 
org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120)
at 
org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553)
at 
org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
at 
org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1261)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1445)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1442)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(Us

[jira] [Created] (HIVE-18384) ConcurrentModificationException in log4j2.x library

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-18384:


 Summary: ConcurrentModificationException in log4j2.x library
 Key: HIVE-18384
 URL: https://issues.apache.org/jira/browse/HIVE-18384
 Project: Hive
  Issue Type: Bug
  Components: Logging
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


In one of the internal testing, observed the following exception
{code}
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) 
~[?:1.8.0_152]
at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[?:1.8.0_152]
at 
java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) 
~[?:1.8.0_152]
at 
org.apache.logging.log4j.message.ParameterFormatter.appendCollection(ParameterFormatter.java:596)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.message.ParameterFormatter.appendPotentiallyRecursiveValue(ParameterFormatter.java:504)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.message.ParameterFormatter.recursiveDeepToString(ParameterFormatter.java:429)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.message.ParameterFormatter.formatMessage2(ParameterFormatter.java:189)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:224)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.message.ParameterizedMessage.getFormattedMessage(ParameterizedMessage.java:200)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.RingBufferLogEvent.setMessage(RingBufferLogEvent.java:126)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.RingBufferLogEvent.setValues(RingBufferLogEvent.java:104)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:56)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:34)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
com.lmax.disruptor.RingBuffer.translateAndPublish(RingBuffer.java:930) 
~[disruptor-3.3.0.jar:?]
at com.lmax.disruptor.RingBuffer.tryPublishEvent(RingBuffer.java:456) 
~[disruptor-3.3.0.jar:?]
at 
org.apache.logging.log4j.core.async.AsyncLoggerDisruptor.tryPublish(AsyncLoggerDisruptor.java:190)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.AsyncLogger.publish(AsyncLogger.java:160) 
~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.AsyncLogger.logWithThreadLocalTranslator(AsyncLogger.java:156)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.core.async.AsyncLogger.logMessage(AsyncLogger.java:126)
 ~[log4j-core-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2011)
 ~[log4j-api-2.6.2.jar:2.6.2]
at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1884)
 ~[log4j-api-2.6.2.jar:2.6.2]
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:189) 
~[log4j-slf4j-impl-2.6.2.jar:2.6.2]
at 
org.apache.hadoop.hive.druid.security.KerberosHttpClient.inner_go(KerberosHttpClient.java:96)
 ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
at 
org.apache.hadoop.hive.druid.security.KerberosHttpClient.access$100(KerberosHttpClient.java:50)
 ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
at 
org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:144)
 ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
at 
org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:134)
 ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
at 
org.apache.hive.druid.com.google.common.util.concurrent.Futures$4.run(Futures.java:1181)
 ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_152]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
{code}

The fix for this went into 2.9.1 LOG4J2-1988 onwards. Updating log4j to latest 
version should have a fix for this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 64900: HIVE-18359: Extend grouping set limits from int to long

[Jesús Camacho Rodríguez]

> On Jan. 5, 2018, 4:32 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFGrouping.java
> > Line 63 (original), 63 (patched)
> > 
> >
> > I think it should be sufficient to check for LONG.
> 
> Prasanth_J wrote:
> The reason I am checking both INT and LONG here is there are many places 
> in compiler that assumes Int for grouping. I fixed in many places, however 
> there are places where string constansts "0" are initialized for 
> "grouping__id" column, some cases where type inferences returns int, some 
> cases constant folding returns int which became too intrusive of a change. So 
> I handled it here in the UDF. Anyway, we use getLong() to retrieve the value. 
> So I guess it is not a problem? As long as the bitmask is returned as long I 
> think it shouldn't be a problem. I will wait for another test run to confirm 
> this statement :)
> 
> Jesús Camacho Rodríguez wrote:
> If only LONG is used, where do you observe the issue / for which tests? I 
> saw that you changed it already in many places, including Calcite, thus it 
> should be fine? If you let me know the test, I can help taking a look too. I 
> agree that it might not be a problem, but I am not sure in which cases we 
> would end up referencing an INT, hence I just want to make sure that we are 
> not masking another issue.
> 
> Prasanth_J wrote:
> These are the test where I observed failures when I changed this case to 
> LONG only
> 
> groupby_grouping_sets_grouping.q,groupby_grouping_window.q,groupby_rollup_empty.q,udf_width_bucket.q,vector_grouping_sets.q
> 
> Some pointers 
> 
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java#L1188
> 
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java#L866
> 
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L3131
> 
> The rabbit hole went a little deep, so I handled it in the UDF.
> 
> Jesús Camacho Rodríguez wrote:
> Thanks for the links Prasanth.
> For the first two, we should not need to change anything as the constant 
> is just a placeholder. For the third, the change should not be in that line, 
> but rather in:
> 
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L3113
>  (maybe a typo in the line number? :) )
> Once you upload the new patch with the fix for grouping func, I will try 
> to repro.
> 
> Prasanth_J wrote:
> I uploaded new patch. groupby_grouping_window.q is still failing.

Thanks Prasanth, I have been checking and this only happens when we enter the 
first branch in the if clause in 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L3113
 , as apparently even with change it is interpreted as INT.
Since that is the only case, I think it is OK to bring back the change to allow 
INT and LONG for the first parameter in the function and push the patch if 
tests pass.
Sorry about the hassle, I just wanted to understand what was going on, as it is 
easy to create errors inadvertently in that part of the code.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64900/#review194829
---


On Jan. 5, 2018, 6:55 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64900/
> ---
> 
> (Updated Jan. 5, 2018, 6:55 a.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-18359
> https://issues.apache.org/jira/browse/HIVE-18359
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18359: Extend grouping set limits from int to long
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8b94d1d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
> 90145e5 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java 0032305 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveGroupingID.java
>  adcda26 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
>  89c5c23 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
>  6f4188c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 28b4cfe 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5

[jira] [Created] (HIVE-18383) Qtests: running all cases from TestNegativeCliDriver results in OOMs

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-18383:
---

 Summary: Qtests: running all cases from TestNegativeCliDriver 
results in OOMs
 Key: HIVE-18383
 URL: https://issues.apache.org/jira/browse/HIVE-18383
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


I think that it is caused by unclosed SessionState objects which are piling up 
and cause OOM..

There is special have been made to start a new sessionstate for every qtest; 
but the old one is not closed up to this 
[point|https://github.com/apache/hive/blob/20c9a3905f4b1b627c935ad54a53a7a59015587c/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L1202]

this prevents running all {{TestNegativeCliDriver}} tests in one maven 
callI keep getting OOMs

This issues sometimes appears on the ptest executor as well and its reported as 
a failed batch.


I've gone back in time a bitseems like at 
c925cf8d2bdf646f5c3c57ed7252c01b2ab33eec it was ok to execute the whole batch; 
but at 1b4baf474c15377cc9f0bacdda317feabeefacaf and probably also at 
a42314deb07a1c8d9d4daeaa799ad1c1ebb0c6c9 its not possible anymore. I suspect 
that there is possibly another issueor these are just the consequences that 
the sessionstate got heavier by a few hundred bytes; and made it easier to fill 
up the heap



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)