[jira] [Created] (HIVE-18759) Remove unconnected q.out-s

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-18759:
---

 Summary: Remove unconnected q.out-s
 Key: HIVE-18759
 URL: https://issues.apache.org/jira/browse/HIVE-18759
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Zoltan Haindrich


org.apache.hadoop.hive.cli.control.TestDanglingQOuts



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #309: HIVE-18754 : REPL STATUS should support 'with' claus...

[maheshk114]

GitHub user maheshk114 opened a pull request:

https://github.com/apache/hive/pull/309

HIVE-18754 : REPL STATUS should support 'with' clause

Added support for WITH clause in REPL STATUS commnad.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maheshk114/hive BUG-96847

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/309.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #309


commit 4899e33200ec116816d9e4a6c97c5e8c17a4ea5d
Author: Mahesh Kumar Behera 
Date:   2018-02-20T09:40:07Z

HIVE-18754 : REPL STATUS should support 'with' clause




---

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/#review197837
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Lines 218 (patched)


This is a short lived temp dir, there is no need to inherit perms for this. 
Actual partition dir perms inherit will be handled in move task.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Line 1155 (original), 1161 (patched)


There is no need for this. This method is called only in operator tree to 
either create temp dirs or files within temp dirs. In either case inheriting 
from parent is useless. Move Task sets permissions correctly. No need for this 
change.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Line 1167 (original), 1172 (patched)


There is no need for this. This method is called only in operator tree to 
either create temp dirs or files within temp dirs. In either case inheriting 
from parent is useless. Move Task sets permissions correctly. No need for this 
change.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2927 (patched)


Please see the comment above in this method. This is not required.


- Ashutosh Chauhan


On Feb. 20, 2018, 8:33 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65663/
> ---
> 
> (Updated Feb. 20, 2018, 8:33 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 2b7a57bb36 
>   common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 16fc96eae9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 6bba057e76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java a9d03d060a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 0282763587 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cf2c2bc48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 614c29bb15 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eefa8f756c 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 5deec4b0f9 
> 
> 
> Diff: https://reviews.apache.org/r/65663/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18758) Vectorization: Fix VectorUDAFVarFinal produces Wrong Results

[Matt McCline (JIRA)]

Matt McCline created HIVE-18758:
---

 Summary: Vectorization: Fix VectorUDAFVarFinal produces Wrong 
Results
 Key: HIVE-18758
 URL: https://issues.apache.org/jira/browse/HIVE-18758
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


Fix and turn back on vectorization for issue found in 
https://issues.apache.org/jira/browse/HIVE-18756



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18757) LLAP IO for text fails for empty files

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18757:
---

 Summary: LLAP IO for text fails for empty files
 Key: HIVE-18757
 URL: https://issues.apache.org/jira/browse/HIVE-18757
 Project: Hive
  Issue Type: Bug
Reporter: Aswathy Chellammal Sreekumar
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18756) Vectorization: VectorUDAFVarFinal produces Wrong Results

[Matt McCline (JIRA)]

Matt McCline created HIVE-18756:
---

 Summary: Vectorization: VectorUDAFVarFinal produces Wrong Results
 Key: HIVE-18756
 URL: https://issues.apache.org/jira/browse/HIVE-18756
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


For a large query.  Disabling vectorization for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18755) Modifications to the metastore for catalogs

[Alan Gates (JIRA)]

Alan Gates created HIVE-18755:
-

 Summary: Modifications to the metastore for catalogs
 Key: HIVE-18755
 URL: https://issues.apache.org/jira/browse/HIVE-18755
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Reporter: Alan Gates
Assignee: Alan Gates


Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: tests on branch-2?

[Vihang Karajgaonkar]

branch-2 HiveQA works as long as you name the patch file with convention
.-.patch You can take a look at HIVE-18449
 for example.

On Tue, Feb 20, 2018 at 12:34 PM, Sergey Shelukhin 
wrote:

> Do we have HiveQA set up for branch-2?
> I noticed that tests were not run on HIVE-18710-branch-2.patch; they have
> run for me in the past on the similarly named branch-2 patches.
>
>

tests on branch-2?

[Sergey Shelukhin]

Do we have HiveQA set up for branch-2?
I noticed that tests were not run on HIVE-18710-branch-2.patch; they have
run for me in the past on the similarly named branch-2 patches.

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/
---

(Updated Feb. 20, 2018, 8:33 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 2b7a57bb36 
  common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 16fc96eae9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
6bba057e76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java a9d03d060a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 0282763587 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cf2c2bc48 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 614c29bb15 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eefa8f756c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
5deec4b0f9 


Diff: https://reviews.apache.org/r/65663/diff/2/

Changes: https://reviews.apache.org/r/65663/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Sergey Shelukhin]

> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
> > Line 225 (original), 227 (patched)
> > 
> >
> > inherit perms feature till now is for insert statements, not for 
> > concat/merge files. If you want to increase the scope of a feature we shall 
> > do it in a seperate jira and limit this jira for acid tables perm 
> > inheritance.

Merge can happen as part of normal query execution, so it is needed for full 
support.
"Fortunately", it's not supported for ACID in 2.X, so I'll just leave a TODO.


> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
> > Line 231 (original), 235 (patched)
> > 
> >
> > Permission inheritance is taken care in MoveTask. Why is there a need 
> > to do this here?

Nm, this would only be needed for MM tables, not in 2.X


> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2928 (patched)
> > 
> >
> > Please read comment above, this is not needed.

Not sure what you mean. As far as I can tell mvFile does not inherit 
permissions in the code


> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 3181 (patched)
> > 
> >
> > Because feature is for both perms and grp.

But it takes the dest status, and also preserves /source/ group (from the 
temporary file)


> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 3182 (original), 3186 (patched)
> > 
> >
> > These are files. No need for recursive here.

Actually for ACID, some cases of union, etc they may not be files. There was a 
recent bug with DML events about that


> On Feb. 15, 2018, 5:38 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 3268 (original), 3278 (patched)
> > 
> >
> > increasing scope of feature.

This is used when loading partitions and tables after the query and as such can 
easily apply to ACID tables. 
lP/lT is called for DP, etc., and it's easier to add it here then to trace all 
possible paths these calls can take.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/#review197625
---


On Feb. 15, 2018, 1:28 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65663/
> ---
> 
> (Updated Feb. 15, 2018, 1:28 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 
> 2b7a57bb369f0fc3204157a15dff761c1e73419d 
>   common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 
> 16fc96eae99705a0625831122afcef3586fe490e 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> 2ed1c6e00417f713ba8ded16b7591973b7e64271 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 6bba057e764099ff2e6ef974577cb6b42fbecab0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 
> 2683f294f61f5e6e1c553c0095fcc11461229c04 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> a9d03d060adeaa5cad6bef48a63c048f23819d01 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 
> 02827635873ffc319fca40fefbb3cea0fdc4c418 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 
> 5cf2c2bc4869fe3f381fd27366a40564eb9c86c2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 
> d2f9e79cd8366a4fa7541660c41fe06d72922b13 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SkewJoinHandler.java 
> cec7c1a8deddd45b44d388e732b45b77fd48a68e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 614c29bb15557e28523d17d6438830dc9b9d359d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> eefa8f756cdcb7d56ed5ff46f4f0bcbfd170a0cf 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 5deec4b0f96682dae9b5558ac26060d845044984 
> 
> 
> Diff: https://reviews.apache.org/r/65663/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> 

Re: Review Request 65422: HIVE-17626

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 20, 2018, 6:41 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3d777f992b 
  itests/src/test/resources/testconfiguration.properties d4f2e539fb 
  ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java d00e639643 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
27b53b8b33 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g e431271d3a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65422/diff/2/

Changes: https://reviews.apache.org/r/65422/diff/1-2/


Testing
---


Thanks,

Zoltan Haindrich

Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18696
https://issues.apache.org/jira/browse/HIVE-18696


Repository: hive-git


Description
---

The idea behind the patch is

1) Separate the partition validation from starting the tasks which create the 
partition folders. 
Instead of doing the checks on the partitions and submit the tasks in one loop, 
separated the validation into a different loop. So first iterate through the 
partitions, validate the table/db names, and check for duplicates. Then if all 
partitions were correct, in the second loop submit the tasks to create the 
partition folders. This way if one of the partitions is incorrect, the 
exception will be thrown in the first loop, before the tasks are submitted. So 
we can be sure that no partition folder will be created if the list contains an 
invalid partition.

2) Handle the exceptions which occur during the execution of the tasks 
differently.
Previously if an exception occured in one task, the remaining tasks were 
canceled, and the newly created partition folders were cleaned up in the 
finally part. The problem was that it could happen that some tasks were still 
not finished with the folder creation when cleaning up the others, so there 
could have been leftover folders. After doing some testing it turned out that 
this use case cannot be avoided completely when canceling the tasks.
The idea of this patch is to set a flag if an exception is thrown in one of the 
tasks. This flag is visible in the tasks and if its value is true, the 
partition folders won't be created. Then iterate through the remaining tasks 
and wait for them to finish. The tasks which are started before the flag got 
set will then finish creating the partition folders. The tasks which are 
started after the flag got set, won't create the partition folders, to avoid 
unnecessary work. This way it is sure that all tasks are finished, when 
entering the finally part where the partition folders are cleaned up.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 f483ca8 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 919ba78 


Diff: https://reviews.apache.org/r/65716/diff/1/


Testing
---

Added some new tests cases to the TestAddPartitions and 
TestAddPartitionsFromPartSpec tests.


Thanks,

Marta Kuczora

Save the date: ApacheCon North America, September 24-27 in Montréal

[Rich Bowen]

Dear Apache Enthusiast,

(You’re receiving this message because you’re subscribed to a user@ or 
dev@ list of one or more Apache Software Foundation projects.)


We’re pleased to announce the upcoming ApacheCon [1] in Montréal, 
September 24-27. This event is all about you — the Apache project community.


We’ll have four tracks of technical content this time, as well as lots 
of opportunities to connect with your project community, hack on the 
code, and learn about other related (and unrelated!) projects across the 
foundation.


The Call For Papers (CFP) [2] and registration are now open. Register 
early to take advantage of the early bird prices and secure your place 
at the event hotel.


Important dates
March 30: CFP closes
April 20: CFP notifications sent
	August 24: Hotel room block closes (please do not wait until the last 
minute)


Follow @ApacheCon on Twitter to be the first to hear announcements about 
keynotes, the schedule, evening events, and everything you can expect to 
see at the event.


See you in Montréal!

Sincerely, Rich Bowen, V.P. Events,
on behalf of the entire ApacheCon team

[1] http://www.apachecon.com/acna18
[2] https://cfp.apachecon.com/conference.html?apachecon-north-america-2018

[jira] [Created] (HIVE-18754) REPL STATUS should support 'with' clause

[mahesh kumar behera (JIRA)]

mahesh kumar behera created HIVE-18754:
--

 Summary: REPL STATUS should support 'with' clause
 Key: HIVE-18754
 URL: https://issues.apache.org/jira/browse/HIVE-18754
 Project: Hive
  Issue Type: Task
  Components: repl, Standalone Metastore
Affects Versions: 3.0.0
Reporter: mahesh kumar behera
 Fix For: 3.0.0


We have support for "WITH" clause in "REPL LOAD" command, but we don't have 
that for "REPL STATUS" command.
With the cloud replication model for DLM 1.1, HiveServer2 is only running in 
the source on-prem cluster.
"REPL LOAD"'s with clause is currently used to pass the remote cloud clusters 
metastore uri, using "hive.metastore.uri" parameter.

Once "REPL LOAD" is run, "REPL STATUS" needs to be run to determine where the 
next incremental replication should start from. Since "REPL STATUS" is also 
going to run on source cluster, we need to add support for the "WITH" clause 
for it.

We should also change the privilege required for "REPL STATUS" command to what 
is required by "REPL LOAD" command as now arbitrary configs can be set for 
"REPL STATUS" using the WITH clause.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65686: HIVE-18728 Secure webHCat with SSL

[Lefty Leverenz]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65686/#review197767
---




hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/AppConfig.java
Lines 113-117 (patched)


These configs look fine to me.  After the patch is committed they will need 
to be documented in  
https://cwiki.apache.org/confluence/display/Hive/WebHCat+Configure#WebHCatConfigure-ConfigurationVariables.
 
HIVE-18728's description has the documentation.


- Lefty Leverenz


On Feb. 16, 2018, 2:19 p.m., Oleksiy Sayankin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65686/
> ---
> 
> (Updated Feb. 16, 2018, 2:19 p.m.)
> 
> 
> Review request for hive, Lefty Leverenz, Lefty Leverenz, Rui Li, Lenni Kuff, 
> Szehon Ho, and Szehon Ho.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Initiall commit
> 
> 
> Diffs
> -
> 
>   
> hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/AppConfig.java
>  1fd9e47 
>   
> hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
>  d183b2e 
> 
> 
> Diff: https://reviews.apache.org/r/65686/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Oleksiy Sayankin
> 
>

[jira] [Created] (HIVE-18753) Correct methods and variables names which uses writeId instead of transactionId.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18753:
---

 Summary: Correct methods and variables names which uses writeId 
instead of transactionId.
 Key: HIVE-18753
 URL: https://issues.apache.org/jira/browse/HIVE-18753
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID implementation (HIVE-18192) have replaced global transaction 
ID with per table write ID to version the data files in a table.

Now, the ACID data files referred by different class methods and variables 
still uses names transactionId to mean write id. 

So, it is required to rename those methods/variables/classes to mean writeId 
instead of transactionId.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18752) HiveEndPoint: Optimise opening batch transactions and getting write Ids for each transaction in the batch into single metastore api.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18752:
---

 Summary: HiveEndPoint: Optimise opening batch transactions and 
getting write Ids for each transaction in the batch into single metastore api.
 Key: HIVE-18752
 URL: https://issues.apache.org/jira/browse/HIVE-18752
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, Metastore
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID implementation (HIVE-18192) have introduced write ID and 
maps it against the txn.

Now, for streaming ingest, we need to open txns batch and then allocate write 
id for each txn in the batch which is 2 metastore calls.

This can be optimised to use only one metastore api.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18751) get_splits UDF on ACID table scan doesn't receive ValidWriteIdList configuration.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18751:
---

 Summary: get_splits UDF on ACID table scan doesn't receive 
ValidWriteIdList configuration.
 Key: HIVE-18751
 URL: https://issues.apache.org/jira/browse/HIVE-18751
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID (HIVE-18192) have replaced global transaction ID with write 
ID to version data files in ACID/MM tables,

To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
txn/table and use it when scan the ACID/MM tables.

In case of get_splits UDF which runs on ACID table scan query won't receive it 
properly through configuration and hence throws exception. 

TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to fix 
it.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18750) Exchange partition should not be supported with per table write ID.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18750:
---

 Summary: Exchange partition should not be supported with per table 
write ID.
 Key: HIVE-18750
 URL: https://issues.apache.org/jira/browse/HIVE-18750
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write id implementation (HIVE-18192) have introduced write ID per 
table and used write ID to name the delta/base files and also as primary key 
for each row.

Now, exchange partition have to move delta/base files across tables without 
changing the write ID which causes incorrect results. 

Also, this exchange partition feature is there to support the use-case of 
atomic updates. But with ACID updates, we shall support atomic-updates and 
hence it makes sense to not support exchange partition for ACID and MM tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18749) Need to replace transactionId to per table writeId in RecordIdentifier.Field.transactionId

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18749:
---

 Summary: Need to replace transactionId to per table writeId in 
RecordIdentifier.Field.transactionId
 Key: HIVE-18749
 URL: https://issues.apache.org/jira/browse/HIVE-18749
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID implementation (HIVE-18192) have replaced global transaction 
ID with write ID for the primary key for a row marked by 
RecordIdentifier.Field..transactionId.

Need to replace the same with writeId and update all test results file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18748) Rename tables should update the table names in NEXT_WRITE_ID and TXN_TO_WRITE_ID tables.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18748:
---

 Summary: Rename tables should update the table names in 
NEXT_WRITE_ID and TXN_TO_WRITE_ID tables. 
 Key: HIVE-18748
 URL: https://issues.apache.org/jira/browse/HIVE-18748
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID implementation (HIVE-18192) introduces couple of metatables 
such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids allocated per 
table.

Now, when we rename any tables, it is necessary to update the corresponding 
table names in these table as well. Otherwise, ACID table operations won't work 
properly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18747) Cleaner for TXNS_TO_WRITE_ID table entries.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18747:
---

 Summary: Cleaner for TXNS_TO_WRITE_ID table entries.
 Key: HIVE-18747
 URL: https://issues.apache.org/jira/browse/HIVE-18747
 Project: Hive
  Issue Type: Sub-task
  Components: repl, Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
and table write ID in TXN_TO_WRITE_ID meta table. 

The entries in this table is used to generate ValidWriteIdList for the given 
ValidTxnList to ensure snapshot isolation. 

When table or database is dropped, then these entries are cleaned-up. But, it 
is necessary to clean-up for active tables too for better performance.

Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which is 
referred by any active ValidTxnList snapshot as open/aborted txn. If no 
references found in this table for any txn, then it is eligible for cleanup.

After clean-up, need to maintain just one entry per table to mark as LWM (low 
water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)