[jira] [Created] (HIVE-18466) Enhance LazySimpleSerDe (Text) to optionally output data escaped for serialization

[Matt McCline (JIRA)]

Matt McCline created HIVE-18466:
---

 Summary: Enhance LazySimpleSerDe (Text) to optionally output data 
escaped for serialization
 Key: HIVE-18466
 URL: https://issues.apache.org/jira/browse/HIVE-18466
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
 Environment: Allows SELECTing out TEXTFILE columns but retaining 
STRING data type family escapes so the output is still TEXTFILE.
Reporter: Matt McCline
Assignee: Matt McCline






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18465) Hive metastore schema initialization failing on postgres

[Deepesh Khandelwal (JIRA)]

Deepesh Khandelwal created HIVE-18465:
-

 Summary: Hive metastore schema initialization failing on postgres
 Key: HIVE-18465
 URL: https://issues.apache.org/jira/browse/HIVE-18465
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal


Hive metastore schema on postgres is broken after the commit for HIVE-14498. 
Following error is seen during schema initialization:
{noformat}
0: jdbc:postgresql://localhost.localdomain:54> ALTER TABLE ONLY "MV_CREATION_MET
ADATA" ADD CONSTRAINT "MV_CREATION_METADATA_FK" FOREIGN KEY ("TBL_ID") REFERENCE
S "TBLS"("TBL_ID") DEFERRABLE
Error: ERROR: there is no unique constraint matching given keys for referenced 
table "TBLS" (state=42830,code=0)
Closing: 0: jdbc:postgresql://localhost.localdomain:5432/hive
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
  at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586)
  at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:559)
  at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1183)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.io.IOException: Schema script failed, errorcode 2
  at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:957)
  at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:935)
  at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:582)
  ... 8 more
*** schemaTool failed ***{noformat}
In the file metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
the ordering of statement
{noformat}
ALTER TABLE ONLY "MV_CREATION_METADATA"
 ADD CONSTRAINT "MV_CREATION_METADATA_FK" FOREIGN KEY ("TBL_ID") REFERENCES 
"TBLS"("TBL_ID") DEFERRABLE;{noformat}
is before the definition of unique constraints for TBLS which is causing the 
issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18464) Fix TestSSL test failures

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-18464:
--

 Summary: Fix TestSSL test failures
 Key: HIVE-18464
 URL: https://issues.apache.org/jira/browse/HIVE-18464
 Project: Hive
  Issue Type: Test
Reporter: Vihang Karajgaonkar


TestSSL test has been failing for a while now. Following three tests fail 
everytime precommit job runs.

org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65185: HIVE-18457 improve show plan output (triggers, mappings)

[Harish Jaiprakash]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65185/#review195530
---



Minor issues, code is fine.


ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
Line 509 (original), 507 (patched)


Earlier code was flat pools array. This gets called in 2 contexts now, both 
rp and pools. It will work for pool under root. But will not work for children 
of pool. We should start another pools/children array inside this.

There is a test case TestJsonRPFormatter, which should be updated for this.



ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
Line 771 (original), 797 (patched)


if p1.pool == null && p2.pool == null then it should return 0;



ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
Lines 883 (patched)


Because of foreign key constraints unusedTriggers and unusedMappings should 
not happen.


- Harish Jaiprakash


On Jan. 17, 2018, 7:45 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65185/
> ---
> 
> (Updated Jan. 17, 2018, 7:45 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 8f28b62311 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  035c5fcc91 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  489842ee82 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  0f1e893de6 
>   ql/src/test/queries/clientpositive/resourceplan.q b46c1ee2d6 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 72a78b1b4a 
> 
> 
> Diff: https://reviews.apache.org/r/65185/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65084: HIVE-18430 Add new determinism category for runtime constants (current_date, current_timestamp)

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65084/#review195531
---




ql/src/java/org/apache/hadoop/hive/ql/udf/UDFType.java
Lines 72 (patched)


1) I think this should be _true_ by default, similar to _deterministic_. 
This change is in Hive 3.0, hence we just need to document it properly, but 
backwards compatibility should not be an issue.

2) _deterministic_ functions should always be _runtimeConstant_, correct? 
Could this be enforced?


- Jesús Camacho Rodríguez


On Jan. 12, 2018, 7:33 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65084/
> ---
> 
> (Updated Jan. 12, 2018, 7:33 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-18430
> https://issues.apache.org/jira/browse/HIVE-18430
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add a runtimeConstant field to UDFType. Also add a new method 
> isConsistentWithinQuery() to FunctionRegistry/ExprNodeEvaluator, which 
> indicates whether the UDF/expression returns a consistent value during the 
> life of the query. Most existing calls to isDeterministic() should be 
> switched to use isConsistentWithinQuery().
> 
> current_timestamp/current_date/current_user/current_database/logged_in_user 
> are now tagged as non-determinstic, and runtime constants.
> Jesus/Ashutosh: How does setting these functions to non-deterministic affect 
> Calcite? It appears that for Calcite operators, there is a 
> isDynamicFunction() which should return true in the case of runtime 
> constants. Within Calcite should isDeterministic() return true or false for 
> these functions?
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java 375d65f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 
> cc40cae 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
> 8b9baa6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 7ca950d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
>  b0a2da8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerExpressionOperatorFactory.java
>  306e714 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  5b72dbd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  a5d950a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/PartitionPrune.java
>  0e5e2b9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  13ee4e5 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 461dbe5 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
> ac37cc4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5a88a96 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 67ea32c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 067fbe0 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java 
> d4df1e8 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java e265863 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFType.java ac3ec58 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java ef8dcf0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentDate.java 
> 1f027a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentTimestamp.java
>  2f13a22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentUser.java 
> d97583d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLoggedInUser.java 
> 2915b86 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
> 3f505f2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFCurrentDB.java a5bab4f 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java 
> 3589938 
> 
> 
> Diff: https://reviews.apache.org/r/65084/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Switched PreCommit-Admin over to Apache Yetus

[Vihang Karajgaonkar]

Thanks Allen for fixing the precommit-admin job.

All,

The issue with precommit hive job reported in
https://issues.apache.org/jira/browse/HIVE-18461 is fixed as well. So
automated precommit tests should be back for Hive. Incase your patch didn't
make the precommit tests during this downtime, you will need to reattach
the patch again so that it gets triggered in the next run.

Thanks,
Vihang


On Tue, Jan 16, 2018 at 6:15 PM, Allen Wittenauer  wrote:

> bcc: bui...@apache.org, d...@hbase.apache.org, dev@hive.apache.org,
> common-...@hadoop.apache.org, d...@phoenix.apache.org, d...@oozie.apache.org
>
> (These are all of the projects that had pending issues.  This is
> not all of the groups that actually rely upon this code...)
>
> The recent JIRA upgrade broke PreCommit-Admin.
>
> This breakage was obviously an unintended consequence.  This
> python code hasn’t been touched in a very long time.  In fact, it was still
> coming from Hadoop SVN tree as it had never been migrated to git.  So it
> isn’t too surprising that after all this time something finally broke it.
>
> Luckily, Apache Yetus was already in the progress of adopting it
> for a variety of reasons that I won't go into here.  With the breakage,
> this work naturally became more urgent.  With the help of the Apache Yetus
> community doing some quick reviews, I just switched PreCommit-Admin over to
> using the master version of the equivalent code in the Yetus source tree.
> As soon as the community can get a 0.7.0 release out, we’ll switch it over
> from master to 0.7.0 so that it can follow our regular release cadence.
> This also means that JIRA issues can be filled against Yetus for bugs seen
> in the code base or for feature requests.  [Hopefully with code and docs
> attached. :) ]
>
> In any case, with the re-activation of this job, all unprocessed
> jobs just kicked off.  So don't be too surprised by the influx of feedback.
>
> As a sidenote, there are some other sticky issues with regards to
> precommit setups on Jenkins.  I'll be sending another note in the future on
> that though. I've had enough excitement for today. :)
>
> Thanks!
>
>

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
---

(Updated Jan. 17, 2018, 2:27 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

Implemented review comments.
Added a test for ACID tables.
Removed the strict ACID check which ensured that files are named in format 
00_0, 00_0_copy_1 etc. The renaming is now done explicitely for 
non-bucketed tables.


Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350


Repository: hive-git


Description
---

Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with 
inconsistent file name.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java d68d646eb4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
4535c3edc2 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
cc1d8574b0 
  ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
  ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
  ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
  ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 


Diff: https://reviews.apache.org/r/65130/diff/4/

Changes: https://reviews.apache.org/r/65130/diff/3-4/


Testing
---


Thanks,

Deepak Jaiswal

Switched PreCommit-Admin over to Apache Yetus

[Allen Wittenauer]

bcc: bui...@apache.org, d...@hbase.apache.org, dev@hive.apache.org, 
common-...@hadoop.apache.org, d...@phoenix.apache.org, d...@oozie.apache.org

(These are all of the projects that had pending issues.  This is not 
all of the groups that actually rely upon this code...)

The recent JIRA upgrade broke PreCommit-Admin.  

This breakage was obviously an unintended consequence.  This python 
code hasn’t been touched in a very long time.  In fact, it was still coming 
from Hadoop SVN tree as it had never been migrated to git.  So it isn’t too 
surprising that after all this time something finally broke it.

Luckily, Apache Yetus was already in the progress of adopting it for a 
variety of reasons that I won't go into here.  With the breakage, this work 
naturally became more urgent.  With the help of the Apache Yetus community 
doing some quick reviews, I just switched PreCommit-Admin over to using the 
master version of the equivalent code in the Yetus source tree.  As soon as the 
community can get a 0.7.0 release out, we’ll switch it over from master to 
0.7.0 so that it can follow our regular release cadence.  This also means that 
JIRA issues can be filled against Yetus for bugs seen in the code base or for 
feature requests.  [Hopefully with code and docs attached. :) ]

In any case, with the re-activation of this job, all unprocessed jobs 
just kicked off.  So don't be too surprised by the influx of feedback.

As a sidenote, there are some other sticky issues with regards to 
precommit setups on Jenkins.  I'll be sending another note in the future on 
that though. I've had enough excitement for today. :)

Thanks!

Review Request 65185: HIVE-18457 improve show plan output (triggers, mappings)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65185/
---

Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
8f28b62311 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 035c5fcc91 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
 489842ee82 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 0f1e893de6 
  ql/src/test/queries/clientpositive/resourceplan.q b46c1ee2d6 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 72a78b1b4a 


Diff: https://reviews.apache.org/r/65185/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18463) Typo in the JdbcConnectionParams constructor

[Oleg Danilov (JIRA)]

Oleg Danilov created HIVE-18463:
---

 Summary: Typo in the JdbcConnectionParams constructor
 Key: HIVE-18463
 URL: https://issues.apache.org/jira/browse/HIVE-18463
 Project: Hive
  Issue Type: Bug
Reporter: Oleg Danilov


Seems like the last one should be params.rejectedHostZnodePaths as well:

 
{code:java}
public JdbcConnectionParams(JdbcConnectionParams params) {
this.host = params.host;
this.port = params.port;
...
this.rejectedHostZnodePaths.addAll(rejectedHostZnodePaths);
}

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65173: HIVE-18386

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65173/#review195524
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2431 (patched)


I wonder whether this config should be called 
hive.server2.materializedviews.registry.impl with possible values of 
dummy,default etc. That way in future id we decide to change registry 
implementation we can reuse this config. 
That likely will also require to define an interface for this. But, that we 
can do later. Right now just changing name and value of config is sufficient, 
so that there are no user visible changes later.



ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
Lines 141 (patched)


LOG.info



ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
Line 249 (original), 282 (patched)


Caller of this method (Loader and Hook) has conf already they can pass it 
in. new HiveConf() is expensive.



ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
Lines 392 (patched)


Caller should pass in conf.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Line 192 (original), 192 (patched)


SessionHive should be well formed object here which can be passed in to 
callee. I see previous method is also using it. Any reason to not do that.


- Ashutosh Chauhan


On Jan. 16, 2018, 10:54 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65173/
> ---
> 
> (Updated Jan. 16, 2018, 10:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Thejas Nair.
> 
> 
> Bugs: HIVE-18386
> https://issues.apache.org/jira/browse/HIVE-18386
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18386
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 
> 2cd3c95dc996671f4b176ab44f8584f039537e5f 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 854bbdf2f10e9c33bc7ad4d5052ea532430fbf01 
>   data/conf/hive-site.xml 9f6aec178d968ee5183ee36824426ea50d3abe62 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 9b102f2a367f04b83e886e231b09f92e6be14088 
>   
> ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryPropertiesHook.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  7e5c81b6fd9a3264579585dd2043a27e7df72b4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  80c7804dc1c40efecb61d28d4bb15da14c427054 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite_dummy.q 
> PRE-CREATION 
>   
> ql/src/test/queries/clientpositive/materialized_view_create_rewrite_rebuild_dummy.q
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/materialized_view_create_rewrite_dummy.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/materialized_view_create_rewrite_rebuild_dummy.q.out
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 
> 58b8fb42f2d4c875333a2257f859d32853292002 
> 
> 
> Diff: https://reviews.apache.org/r/65173/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Deepak Jaiswal]

> On Jan. 16, 2018, 11:06 p.m., Jason Dere wrote:
> > - Why the need for the file name change even for non-bucketed tables?
> > - With this patch, what happens to existing tables (bucketed and 
> > non-bucketed) which contain filenames that do not match the format used 
> > here?
> > - How are external tables handled, where the user is responsible for 
> > managing the files in the table directories? For both bucketed and 
> > non-bucketed tables.

Will update the JIRA with details.


> On Jan. 16, 2018, 11:06 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 3951 (patched)
> > 
> >
> > Add a comment about the assumptions/actions taken here - it looks like 
> > bucketed files are assumed to be in the correct name format? Is there any 
> > validation that the 
> > 
> > And all non-bucketed files are renamed to 00_0? I wonder if this 
> > will ever cause a problem with tons of files with 00_0, 
> > 00_0_copy_1, 00_0_copy_2 etc. Though I guess this is currently what 
> > happens when INSERT INTO TABLE occurs.

Bucketed files are validated in LoadSemanticAnalyzer itself. However, I can 
certainly add an assert for that. Thanks for pointing this out.
Will add a comment as well.


> On Jan. 16, 2018, 11:06 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 4034 (patched)
> > 
> >
> > This check looks ugly, but if you have to do it then maybe do what is 
> > done in SemanticAnalyzer.analyzeCreateTable() - 
> > .startsWith(HiveConf.getVar(conf, HiveConf.ConfVars.STAGINGDIR))
> > 
> > Also, you are invoking toString() on an array (srcs) - use 
> > srcs[0].getPath().getName().
> > 
> > Also please add a comment here to describe why this extra check is here.

Yes it sure does. I will use the example to make it better. Will add a comment 
as well.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/#review195503
---


On Jan. 16, 2018, 9:31 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65130/
> ---
> 
> (Updated Jan. 16, 2018, 9:31 a.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-18350
> https://issues.apache.org/jira/browse/HIVE-18350
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made changes for both bucketed and non-bucketed tables.
> Added a positive test for non-bucketed table which renames the loaded file.
> Added couple of negative tests for bucketed table which reject a load with 
> inconsistent file name.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 4535c3edc2 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
> cc1d8574b0 
>   ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
>   ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
>   ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
>   ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 
> 
> 
> Diff: https://reviews.apache.org/r/65130/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-18462) Explain formatted for queries with map join has columnExprMap with unformatted column name

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18462:
--

 Summary: Explain formatted for queries with map join has 
columnExprMap with unformatted column name
 Key: HIVE-18462
 URL: https://issues.apache.org/jira/browse/HIVE-18462
 Project: Hive
  Issue Type: Improvement
Reporter: Vineet Garg
Assignee: Vineet Garg


e.g.

{code:sql}
"columnExprMap:":{  
  "_col0":"0:Column[_col0]",
  "_col1":"0:Column[_col1]",
  "_col2":"1:Column[_col0]",
  "_col3":"1:Column[_col1]"
   }
{code}

It is better formatted as:

"columnExprMap:":{  
  "_col0":"0:_col0",
  "_col1":"0:_col1",
  "_col2":"1:_col0",
  "_col3":"1:_col1"
   }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18461) Fix precommit hive job

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-18461:
--

 Summary: Fix precommit hive job
 Key: HIVE-18461
 URL: https://issues.apache.org/jira/browse/HIVE-18461
 Project: Hive
  Issue Type: Task
  Components: Testing Infrastructure
Reporter: Vihang Karajgaonkar


JIRA was upgraded over the weekend and precommit job has been failing since 
then. There are potentially two issues at play here. One is with the precommit 
admin job which automate the patch testing. I think YETUS-594 should fix the 
precommit admin job. But manually submission of Hive jobs is failing with below 
exception. We should get this fix to get the automated testing back on track.

{noformat}
+ local 
'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
+ java -cp 
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
 org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
 --password '[***]' --testHandle PreCommit-HIVE-Build-8631 --endpoint 
http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint 
http://104.198.109.242/logs/ --profile master-mr2 --patch 
https://issues.apache.org/jira/secure/attachment/12906251/HIVE-18323.05.patch 
--jira HIVE-18323
Exception in thread "main" javax.net.ssl.SSLException: Received fatal alert: 
protocol_version
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1979)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1086)
at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
at 
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at 
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
at 
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at java.net.URL.openStream(URL.java:1041)
at 
com.google.common.io.Resources$UrlByteSource.openStream(Resources.java:72)
at com.google.common.io.ByteSource.read(ByteSource.java:257)
at com.google.common.io.Resources.toByteArray(Resources.java:99)
at 
org.apache.hive.ptest.api.client.PTestClient.testStart(PTestClient.java:126)
at 
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:320)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Rebasing standalone-metastore branch

[Vihang Karajgaonkar]

Thanks for the headsup Alan. I am okay with the rebasing
standalone-metastore branch. Will update my WIP patches accordingly.

On Tue, Jan 16, 2018 at 2:01 PM, Alan Gates  wrote:

> I’ve committed HIVE-17982 to master.  I have been rebasing
> standalone-metastore after each push of a patch from it to master to keep
> it close to master.  Vihang and maybe Peter have patches against it, so
> this will bust your history.  It will force you to rebase your branches
> when you fetch from Apache.  I don’t think this is any worse than the
> rebase you’d have to do to get those patches on master anyway.  Are you
> guys ok with me doing the rebase?
>
> Alan.
>

[GitHub] hive pull request #292: HIVE-17331: Use Path instead of String as key type o...

[dosoft]

GitHub user dosoft opened a pull request:

https://github.com/apache/hive/pull/292

HIVE-17331: Use Path instead of String as key type of the pathToAliases



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dosoft/hive HIVE-17331

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #292


commit 661897e01fe48a0f80c3ee4e9168667b7d926ba9
Author: Oleg Danilov 
Date:   2017-08-16T10:34:39Z

HIVE-17331: Use Path instead of String as key type of the pathToAliases




---

[GitHub] hive pull request #233: HIVE-17331: Use Path instead of String as key type o...

[dosoft]

Github user dosoft closed the pull request at:

https://github.com/apache/hive/pull/233


---

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/#review195503
---



- Why the need for the file name change even for non-bucketed tables?
- With this patch, what happens to existing tables (bucketed and non-bucketed) 
which contain filenames that do not match the format used here?
- How are external tables handled, where the user is responsible for managing 
the files in the table directories? For both bucketed and non-bucketed tables.


ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3951 (patched)


Add a comment about the assumptions/actions taken here - it looks like 
bucketed files are assumed to be in the correct name format? Is there any 
validation that the 

And all non-bucketed files are renamed to 00_0? I wonder if this will 
ever cause a problem with tons of files with 00_0, 00_0_copy_1, 
00_0_copy_2 etc. Though I guess this is currently what happens when INSERT 
INTO TABLE occurs.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 4034 (patched)


This check looks ugly, but if you have to do it then maybe do what is done 
in SemanticAnalyzer.analyzeCreateTable() - .startsWith(HiveConf.getVar(conf, 
HiveConf.ConfVars.STAGINGDIR))

Also, you are invoking toString() on an array (srcs) - use 
srcs[0].getPath().getName().

Also please add a comment here to describe why this extra check is here.


- Jason Dere


On Jan. 16, 2018, 9:31 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65130/
> ---
> 
> (Updated Jan. 16, 2018, 9:31 a.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-18350
> https://issues.apache.org/jira/browse/HIVE-18350
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made changes for both bucketed and non-bucketed tables.
> Added a positive test for non-bucketed table which renames the loaded file.
> Added couple of negative tests for bucketed table which reject a load with 
> inconsistent file name.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 4535c3edc2 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
> cc1d8574b0 
>   ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
>   ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
>   ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
>   ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 
> 
> 
> Diff: https://reviews.apache.org/r/65130/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 65173: HIVE-18386

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65173/
---

(Updated Jan. 16, 2018, 10:54 p.m.)


Review request for hive, Ashutosh Chauhan and Thejas Nair.


Bugs: HIVE-18386
https://issues.apache.org/jira/browse/HIVE-18386


Repository: hive-git


Description
---

HIVE-18386


Diffs (updated)
-

  cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 
2cd3c95dc996671f4b176ab44f8584f039537e5f 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
854bbdf2f10e9c33bc7ad4d5052ea532430fbf01 
  data/conf/hive-site.xml 9f6aec178d968ee5183ee36824426ea50d3abe62 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
9b102f2a367f04b83e886e231b09f92e6be14088 
  
ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryPropertiesHook.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 7e5c81b6fd9a3264579585dd2043a27e7df72b4e 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
80c7804dc1c40efecb61d28d4bb15da14c427054 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_dummy.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_rebuild_dummy.q
 PRE-CREATION 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_dummy.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_rebuild_dummy.q.out
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 
58b8fb42f2d4c875333a2257f859d32853292002 


Diff: https://reviews.apache.org/r/65173/diff/2/

Changes: https://reviews.apache.org/r/65173/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[GitHub] hive pull request #291: HIVE-17983 Make the standalone metastore generate ta...

[alanfgates]

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/291

HIVE-17983 Make the standalone metastore generate tarballs etc.

See JIRA for full comments.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17983

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #291


commit 1ba9b62d9ef488355e1a97dbc7237c1472349a24
Author: Alan Gates 
Date:   2017-10-19T23:49:38Z

HIVE-17983 Make the standalone metastore generate tarballs etc.




---

Rebasing standalone-metastore branch

[Alan Gates]

I’ve committed HIVE-17982 to master.  I have been rebasing
standalone-metastore after each push of a patch from it to master to keep
it close to master.  Vihang and maybe Peter have patches against it, so
this will bust your history.  It will force you to rebase your branches
when you fetch from Apache.  I don’t think this is any worse than the
rebase you’d have to do to get those patches on master anyway.  Are you
guys ok with me doing the rebase?

Alan.

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

[Sergey Shelukhin]

> On Jan. 16, 2018, 9:40 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1381 (patched)
> > 
> >
> > Default should be false until Hadoop bug is fixed, since otherwise we 
> > will hit bug in default config.

The bug is not related to the codec pool, it happens with pool off. I added 
this because we are adding new untested (practiclaly speaking) functionality 
with the pool, so it's good to have an option to disable it


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/#review195504
---


On Jan. 13, 2018, 3:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65148/
> ---
> 
> (Updated Jan. 13, 2018, 3:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  1e0eccfa34 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 627e617d91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> 203ef69cf6 
> 
> 
> Diff: https://reviews.apache.org/r/65148/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-18460:
-

 Summary: Compactor doesn't pass Table properties to the Orc writer
 Key: HIVE-18460
 URL: https://issues.apache.org/jira/browse/HIVE-18460
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


 

 CompactorMap.getWrite()/getDeleteEventWriter() both do 
AcidOutputFormat.Options.tableProperties() but
OrcOutputFormat.getRawRecordWriter() does
{noformat}
final OrcFile.WriterOptions opts =
OrcFile.writerOptions(options.getConfiguration());
{noformat}
which ignores tableProperties value.
It should do 
{noformat}
final OrcFile.WriterOptions opts =
OrcFile.writerOptions(options.getTableProperties(), options.getConfiguration());
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: checkstyle changes

[Zoltan Haindrich]

Well okay..but I still feel that checking for 100 kinda takes the value of 
these checks away; because people will just ignore the checks because they are 
red all the timeso I still think it should be raised to some higher value.
Would 120 be ok?
Choosing a higher linelength does not meab that everyone should use that; I 
think its free to use smaller settings.

cheers,
Zoltan


On 8 Dec 2017 4:16 a.m., Rui Li  wrote:
I also believe 140 is a little too long.

BTW do we use 2 or 4 chars for continuation indent? I personally prefer 4,
but I do find both cases in out code.

On Fri, Dec 8, 2017 at 6:20 AM, Alexander Kolbasov 
wrote:

> Problem with 140-wide code isn't just laptops - in many cases we need to do
> side-by-side diffs (e.g. for code reviews) and this doubles the required
> size.
>
> - Alex.
>
> On Thu, Dec 7, 2017 at 1:38 PM, Sergey Shelukhin 
> wrote:
>
> > I think the 140-character change will make the code hard to use on a
> > laptop without a monitor.
> >
> >
> > On 17/12/7, 02:43, "Peter Vary"  wrote:
> >
> > >Disclaimer: I did not have time to test it out, but according to
> > >http://checkstyle.sourceforge.net/config_misc.html#Indentation
> > >
> > >Maybe the indentation could be solved by:
> > >lineWrappingIndentation=2 (default 4)
> > >forceStrictCondition=false (default false)
> > >
> > >http://checkstyle.sourceforge.net/config_misc.html#TrailingComment
> > >
> > >might help with the comments
> > >
> > >Sorry for not being more helpful. Maybe sometime later I will have time
> > >to check these out.
> > >
> > >Thanks,
> > >Peter
> > >
> > >> On Dec 7, 2017, at 10:26 AM, Zoltan Haindri > > 
> > >> >> wrote:
> > >>
> > >> Hello Eugene!
> > >>
> > >> I've looked into doing something with these; but I was not able to
> > >>relieve the warnings you've mentioned:
> > >>
> > >> * the ;// is seems to be not configurable
> > >>   It seems like its handled by the whitespaceafter module; I'm not
> sure
> > >>how to allow / after ;
> > >> * I think that indentation of 4 for many method arguments makes it
> more
> > >>readable; so I think it would be the best to just drop this check...but
> > >>I've not seen any way to do this(w/o disabling the whole indentation
> > >>module...)
> > >>
> > >> maybe someone else should take a look at itI find it pretty hard
> to
> > >>get docs about specific chechkstyle configurations; since the sear > > 
> > keywords mostly contain keywords like: semicolon, whitespace,
> > >>comment...which tends to pull in all kind of garbage results :)
> > >>
> > >> cheers,
> > >> Zoltan
> > >>
> > >> On 6 Dec 2017 8:53 p.m., Eugene Koifman 
> > >>wrote:
> > >> It currently complains about no space between ; and // as in
> “…);//foo”
> > >>
> > >> And also about indentation when a single method call is split into
> > >>multiple lines.
> > >> It insists on 4 chars in this case, though we use 2 in (all?) other
> > >>cases.
> > >>
> > >> Could this be dialed down as well?
> > >>
> > >>
> > >> On 12/5/17, 7:26 AM, "Peter Vary"  wrote:
> > >>
> > >>+1 for the changes
> > >>
> > >>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich  wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I've filed a ticket to make the checkstyle warnings less noisy
> > >>>(https://issues.apache.org/jira/browse/HIVE-18222)
> > >>>
> > >>> * set maxlinelength to 140
> > >>>   I think everyone is working with big-enough displays to handle this
> > >>>:)
> > >>>   There are many methods which have complicated names / arguments /
> > >>>etc ; breaking the lines more frequently hurts readability...
> > >>> * disabled some restrictions like: declaration via get/set
> > >>>methods for protected/package fields are not mandatory
> > >>>
> > >>> If you don't feel comfortable with these changes, please share your
> > >>>point of view.
> > >>>
> > >>> cheers,
> > >>> Zoltan
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
>



--
Best regards!
Rui Li

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/#review195504
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1381 (patched)


Default should be false until Hadoop bug is fixed, since otherwise we will 
hit bug in default config.


- Ashutosh Chauhan


On Jan. 13, 2018, 3:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65148/
> ---
> 
> (Updated Jan. 13, 2018, 3:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  1e0eccfa34 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 627e617d91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> 203ef69cf6 
> 
> 
> Diff: https://reviews.apache.org/r/65148/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[GitHub] hive pull request #279: HIVE-17982 Move metastore specific itests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/hive/pull/279


---

[jira] [Created] (HIVE-18459) hive-exec.jar leaks contents fb303.jar into classpath

[Naveen Gangam (JIRA)]

Naveen Gangam created HIVE-18459:


 Summary: hive-exec.jar leaks contents fb303.jar into classpath
 Key: HIVE-18459
 URL: https://issues.apache.org/jira/browse/HIVE-18459
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
 Environment: thrift classes are now in the hive classpath in the 
hive-exec.jar (HIVE-11553). This makes it hard to test with other versions of 
this library. This library is already a declared dependency and is not required 
to be included in the hive-exec.jar.

I am proposing that we not include these classes like we have done in the past 
releases.
Reporter: Naveen Gangam
Assignee: Naveen Gangam






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: ptest broken?

[Vihang Karajgaonkar]

Okay looks like we need to move from SOAP to REST and change the pre-commit
admin job to make it work (check comment on the infra JIRA above). Does
anyone know how/who can fix that? I don't have permissions to edit the job.

On Tue, Jan 16, 2018 at 11:28 AM, Vihang Karajgaonkar 
wrote:

> Created https://issues.apache.org/jira/browse/INFRA-15849 Hopefully
> someone who manages this job takes a look soon.
>
> On Tue, Jan 16, 2018 at 11:19 AM, Vihang Karajgaonkar  > wrote:
>
>> Looks like the pre-commit admin job which automatically triggers
>> pre-commit hive jobs is broken and has been failing with the following
>> error:
>>
>> + python 
>> /home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py
>>  
>> '--jira-filter=https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12323182/SearchRequest-12323182.xml?tempMax=50'
>>  --jenkins-url=https://builds.apache.org --jenkins-token=hadoopqa --live
>> Traceback (most recent call last):
>>   File 
>> "/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
>>  line 113, in 
>> rssData = httpGet(options.jiraFilter)
>>   File 
>> "/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
>>  line 41, in httpGet
>> return urllib2.urlopen(resource).read()
>>   File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
>> return _opener.open(url, data, timeout)
>>   File "/usr/lib/python2.7/urllib2.py", line 410, in open
>> response = meth(req, response)
>>   File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
>> 'http', request, response, code, msg, hdrs)
>>   File "/usr/lib/python2.7/urllib2.py", line 448, in error
>> return self._call_chain(*args)
>>   File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
>> result = func(*args)
>>   File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
>> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
>> urllib2.HTTPError: HTTP Error 400: 400
>> Build step 'Execute shell' marked build as failure
>> Archiving artifacts
>> [Fast Archiver] Compressed 99.45 KB of artifacts by 96.5% relative to #329112
>> Sending e-mails to: gkesa...@apache.org ni...@apache.org t...@apache.org
>> Finished: FAILURE
>>
>>
>>
>>
>> On Tue, Jan 16, 2018 at 10:54 AM, Zoltan Haindrich <
>> zhaindr...@hortonworks.com> wrote:
>>
>>> Hello,
>>>
>>> I think the triggering mechanism is broken; queuing manually works
>>>
>>> cheers,
>>> Zoltan
>>>
>>> On 16 Jan 2018 7:38 p.m., Deepak Jaiswal 
>>> wrote:
>>> Hi,
>>>
>>> I attached a patch once last night and then again in the morning but no
>>> run triggered after that.
>>> There is no queue of pending runs as well.
>>>
>>> Regards,
>>> Deepak
>>>
>>>
>>
>

[jira] [Created] (HIVE-18458) Workload manager initializes even when interactive queue is not set

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-18458:


 Summary: Workload manager initializes even when interactive queue 
is not set
 Key: HIVE-18458
 URL: https://issues.apache.org/jira/browse/HIVE-18458
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Workload manager gets initialized even when interactive queue is not defined 
(however there is an active resource plan in metastore). Active resource plan 
is used for tez in this case. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18457) Triggers in unmanaged pools are not shown

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-18457:


 Summary: Triggers in unmanaged pools are not shown
 Key: HIVE-18457
 URL: https://issues.apache.org/jira/browse/HIVE-18457
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


Did the following sequence to add triggers to UNMANAGED. I can see the triggers 
added to metastore by IS_IN_UNAMANGED flag is not set in metastore. Also show 
resource plans does not show triggers in unmanaged pool.
{code}
0: jdbc:hive2://localhost:1> show resource plans;
+--+--++
| rp_name  |  status  | query_parallelism  |
+--+--++
| global   | ACTIVE   | NULL   |
| llap | ENABLED  | NULL   |
+--+--++

0: jdbc:hive2://localhost:1>ALTER RESOURCE PLAN llap ACTIVATE;
0: jdbc:hive2://localhost:1>ALTER RESOURCE PLAN global DISABLE;
0: jdbc:hive2://localhost:1>CREATE TRIGGER global.highly_parallel WHEN 
TOTAL_TASKS > 40 DO KILL;
0: jdbc:hive2://localhost:1>ALTER TRIGGER global.highly_parallel ADD TO 
UNMANAGED;

0: jdbc:hive2://localhost:1>CREATE TRIGGER global.big_hdfs_read WHEN 
HDFS_BYTES_READ > 30 DO KILL;
0: jdbc:hive2://localhost:1>ALTER TRIGGER global.big_hdfs_read ADD TO 
UNMANAGED;

0: jdbc:hive2://localhost:1>CREATE TRIGGER global.slow_query WHEN 
EXECUTION_TIME > 10 DO KILL;
0: jdbc:hive2://localhost:1>ALTER TRIGGER global.slow_query ADD TO 
UNMANAGED;

0: jdbc:hive2://localhost:1>CREATE TRIGGER global.some_spills WHEN 
SPILLED_RECORDS > 10 DO KILL;
0: jdbc:hive2://localhost:1>ALTER TRIGGER global.some_spills ADD TO 
UNMANAGED;

0: jdbc:hive2://localhost:1>ALTER RESOURCE PLAN global ENABLE;
0: jdbc:hive2://localhost:1>ALTER RESOURCE PLAN global ACTIVATE;

0: jdbc:hive2://localhost:1> show resource plan global;
++
|line|
++
| global[status=ACTIVE,parallelism=null,defaultPool=default] |
| default[allocFraction=1.0,schedulingPolicy=null,parallelism=4] |
++
{code}

{code:title=mysql}
mysql> select * from wm_trigger;
++---+-+--+---+-+
| TRIGGER_ID | RP_ID | NAME| TRIGGER_EXPRESSION   | 
ACTION_EXPRESSION | IS_IN_UNMANAGED |
++---+-+--+---+-+
| 29 | 1 | highly_parallel | TOTAL_TASKS > 40 | KILL
  ||
| 33 | 1 | big_hdfs_read   | HDFS_BYTES_READ > 30 | KILL
  ||
| 34 | 1 | slow_query  | EXECUTION_TIME > 10  | KILL
  ||
| 35 | 1 | some_spills | SPILLED_RECORDS > 10 | KILL
  ||
++---+-+--+---+-+
{code}

>From the above mysql table, IS_IN_UNMANAGED is not set and 'show resource plan 
>global' is not showing triggers defined in unmanaged pool. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/#review195495
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
Lines 1919 (patched)


this should be removed


- Sergey Shelukhin


On Jan. 13, 2018, 3:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65148/
> ---
> 
> (Updated Jan. 13, 2018, 3:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  1e0eccfa34 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 627e617d91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> 203ef69cf6 
> 
> 
> Diff: https://reviews.apache.org/r/65148/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: ptest broken?

[Vihang Karajgaonkar]

Created https://issues.apache.org/jira/browse/INFRA-15849 Hopefully someone
who manages this job takes a look soon.

On Tue, Jan 16, 2018 at 11:19 AM, Vihang Karajgaonkar 
wrote:

> Looks like the pre-commit admin job which automatically triggers
> pre-commit hive jobs is broken and has been failing with the following
> error:
>
> + python 
> /home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py
>  
> '--jira-filter=https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12323182/SearchRequest-12323182.xml?tempMax=50'
>  --jenkins-url=https://builds.apache.org --jenkins-token=hadoopqa --live
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
>  line 113, in 
> rssData = httpGet(options.jiraFilter)
>   File 
> "/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
>  line 41, in httpGet
> return urllib2.urlopen(resource).read()
>   File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
> return _opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 410, in open
> response = meth(req, response)
>   File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
> 'http', request, response, code, msg, hdrs)
>   File "/usr/lib/python2.7/urllib2.py", line 448, in error
> return self._call_chain(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
> result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> urllib2.HTTPError: HTTP Error 400: 400
> Build step 'Execute shell' marked build as failure
> Archiving artifacts
> [Fast Archiver] Compressed 99.45 KB of artifacts by 96.5% relative to #329112
> Sending e-mails to: gkesa...@apache.org ni...@apache.org t...@apache.org
> Finished: FAILURE
>
>
>
>
> On Tue, Jan 16, 2018 at 10:54 AM, Zoltan Haindrich <
> zhaindr...@hortonworks.com> wrote:
>
>> Hello,
>>
>> I think the triggering mechanism is broken; queuing manually works
>>
>> cheers,
>> Zoltan
>>
>> On 16 Jan 2018 7:38 p.m., Deepak Jaiswal 
>> wrote:
>> Hi,
>>
>> I attached a patch once last night and then again in the morning but no
>> run triggered after that.
>> There is no queue of pending runs as well.
>>
>> Regards,
>> Deepak
>>
>>
>

Re: ptest broken?

[Vihang Karajgaonkar]

Looks like the pre-commit admin job which automatically triggers pre-commit
hive jobs is broken and has been failing with the following error:

+ python 
/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py
'--jira-filter=https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12323182/SearchRequest-12323182.xml?tempMax=50'
--jenkins-url=https://builds.apache.org --jenkins-token=hadoopqa
--live
Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
line 113, in 
rssData = httpGet(options.jiraFilter)
  File 
"/home/jenkins/jenkins-slave/workspace/PreCommit-Admin/nightly/jenkinsPrecommitAdmin.py",
line 41, in httpGet
return urllib2.urlopen(resource).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: 400
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] Compressed 99.45 KB of artifacts by 96.5% relative to #329112
Sending e-mails to: gkesa...@apache.org ni...@apache.org t...@apache.org
Finished: FAILURE




On Tue, Jan 16, 2018 at 10:54 AM, Zoltan Haindrich <
zhaindr...@hortonworks.com> wrote:

> Hello,
>
> I think the triggering mechanism is broken; queuing manually works
>
> cheers,
> Zoltan
>
> On 16 Jan 2018 7:38 p.m., Deepak Jaiswal  wrote:
> Hi,
>
> I attached a patch once last night and then again in the morning but no
> run triggered after that.
> There is no queue of pending runs as well.
>
> Regards,
> Deepak
>
>

Re: ptest broken?

[Zoltan Haindrich]

Hello,

I think the triggering mechanism is broken; queuing manually works

cheers,
Zoltan

On 16 Jan 2018 7:38 p.m., Deepak Jaiswal  wrote:
Hi,

I attached a patch once last night and then again in the morning but no run 
triggered after that.
There is no queue of pending runs as well.

Regards,
Deepak

ptest broken?

[Deepak Jaiswal]

Hi,

I attached a patch once last night and then again in the morning but no run 
triggered after that.
There is no queue of pending runs as well.

Regards,
Deepak

[jira] [Created] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly

[Andrew Sherman (JIRA)]

Andrew Sherman created HIVE-18456:
-

 Summary: Add some tests for HIVE-18367 to check that the table 
information contains the query correctly
 Key: HIVE-18456
 URL: https://issues.apache.org/jira/browse/HIVE-18456
 Project: Hive
  Issue Type: Bug
Reporter: Andrew Sherman
Assignee: Andrew Sherman


This cannot be tested with a CliDriver test so add a java test to check the 
output of 'describe extended', which is changed by HIVE-18367 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18455) Unable to create operation log session directory when parent directory is deleted

[Sean Roberts (JIRA)]

Sean Roberts created HIVE-18455:
---

 Summary: Unable to create operation log session directory when 
parent directory is deleted
 Key: HIVE-18455
 URL: https://issues.apache.org/jira/browse/HIVE-18455
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Sean Roberts
Assignee: Daniel Dai
 Fix For: 1.3.0, 2.0.0


We see intermittent error in hiveserver2 log:
{code}
(HiveSessionImpl.java:setOperationLogSessionDir(211)) - Unable to create 
operation log session directory:xx
{code}
User are not able to retrieve their operation log through Hue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18454) Incorrect rownum estimation in joins

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-18454:
---

 Summary: Incorrect rownum estimation in joins
 Key: HIVE-18454
 URL: https://issues.apache.org/jira/browse/HIVE-18454
 Project: Hive
  Issue Type: Sub-task
  Components: Statistics
Reporter: Zoltan Haindrich


I've seen this probably earlier...row counts seems to be off the 
charts...12 rows estimated when the table has only 10 rows

{code}
create table s (x int);

insert into s values
(1),(2),(3),(4),(5),
(6),(7),(8),(9),(10);

create table tu(id_uv int,id_uw int,u int);
create table tv(id_uv int,v int);
create table tw(id_uw int,w int);

from s
insert overwrite table tu
select x,x,x 
where x<=6 or x=10
insert overwrite table tv
select x,x  
where x<=3 or x=10
insert overwrite table tw
select x,x  
;

set hive.explain.user=true;

explain analyze
select sum(u*v*w) from tu
join tv on (tu.id_uv=tv.id_uv)
join tw on (tu.id_uw=tw.id_uw)
where w>9 and u>1 and v>3;

desc formatted tv;
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
---

(Updated Jan. 16, 2018, 9:31 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

The logic to rename non-bucketed tables should only apply for load data.
That was not happening in previous patch.


Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350


Repository: hive-git


Description
---

Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with 
inconsistent file name.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
4535c3edc2 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
cc1d8574b0 
  ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
  ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
  ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
  ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 


Diff: https://reviews.apache.org/r/65130/diff/3/

Changes: https://reviews.apache.org/r/65130/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal