Re: External vs. Managed Tables - Current state

[Lefty Leverenz]

Agreed, the Hive wiki is woefully incomplete.  But it does mention DROP for
external tables in two sections of the DDL doc:

   - External Tables
   
:
"When dropping an EXTERNAL table, data in the table is NOT deleted from
   the file system."
   - Drop Table
   
:
"When dropping an EXTERNAL table, data in the table will NOT be deleted
   from the file system."

A wiki page comparing managed and external tables would be very helpful,
and the list is a good start.  I suggest you open a JIRA issue.

Who can help Lars with this task?

-- Lefty


On Thu, Jul 19, 2018 at 3:04 PM Lars Francke  wrote:

> Hi,
>
> I've been wondering if anyone can tell me what the differences and
> limitations of managed vs. external tables are these days.
>
> I've seen the docs from Hortonworks[1] that list these features as not
> supported by External tables:
> * Query cache
> * Materialized views, except in a limited way
> * Default statistics gathering
> * Compute queries using statistics
> * Automatic runtime filtering
> * File merging after insert
>
> But there are no details. For some of them I'm not even sure what they are
> supposed to mean because I couldn't find any documentation in our Wiki :(
>
> So I'd love to create a Wiki page that describes the differences between
> managed & external tables and list relevant JIRAs but I need your help for
> that.
>
> I scanned the code for all references to TableType and this is the list I
> found but I'm almost certainly missing something:
>
> * ARCHIVE/UNARCHIVE - DDLTask - Only works for managed tables
> * TRUNCATE - DDLSemanticAnalyzer - Only works for managed tables
> * MERGE/CONCATENATE - HiveRelOpMaterializationValidator - Only works for
> managed tables
> * Constraints - DDLSemanticAnalyzer -  (NOT NULL, DEFAULT, CHECK, only RELY
> ist allowed)
> * IMPORT - ImportSemanticAnalyzer - This has some wild restrictions I
> didn't follow for external tables
> * Query Results Caching - https://issues.apache.org/jira/browse/HIVE-18513
> SemanticAnalyzer - Documentation missing for Results Cache
>
> So there's a bunch of those from the Hortonworks list missing. If anyone
> could point me to documentation or code for those that'd be great.
>
> I also didn't find the most obvious/oldest one: Data is not deleted for
> EXTERNAL tables on DROP, I'm sure I just missed it.
>
> And last but not least: I have not looked at the grammar to see what's
> already forbidden at that level.
>
> As I said: Any hints would be greatly appreciated.
>
> Thank you!
>
> Lars
>
> A side note: There are so many fantastic features in Hive but because lots
> of them are not or under-documented most people I know just stick to the
> basics. That's a pity :(
>
> [1] <
>
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/using-hiveql/content/hive_hive_3_tables.html
> >
>

Re: Review Request 67974: HIVE-20164

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67974/
---

(Updated July 20, 2018, 11:10 p.m.)


Review request for hive, Gopal V and Jason Dere.


Changes
---

Implemented review comments.


Bugs: HIVE-20164
https://issues.apache.org/jira/browse/HIVE-20164


Repository: hive-git


Description
---

Murmur Hash : Make sure CTAS and IAS use correct bucketing version


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties d5a33bd8ca 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 1661aeccd7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java bbce940c2e 
  ql/src/test/queries/clientpositive/murmur_hash_migration.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/murmur_hash_migration.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/67974/diff/2/

Changes: https://reviews.apache.org/r/67974/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20219) analyze and analyze for columns don't manage stats state correctly

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20219:
---

 Summary: analyze and analyze for columns don't manage stats state 
correctly
 Key: HIVE-20219
 URL: https://issues.apache.org/jira/browse/HIVE-20219
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Step 1. Have a table with fully correct stats.
Step 2. Run some query that sets stats to invalid.
Step 3. Run analyze without for columns...

Result is that while only basic stats are updated, the flag is set to true with 
old column stats values.
I blame json.
Not sure if this would affect master, I'll add a test; ACID stats invalidation 
definitely needs to invalidate all the stats, not just basic stats flag, when 
invalidating.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Apache Hive 3.1.0 Release Candidate 0

[Vineet Garg]

My understanding was that standalone-metastore release is to be done 
independently of hive release. Once we verify that standalone-metastore works 
for branch-3.1 we can start the release process for standalone-metastore.

@Thejas,
Can you please confirm this?

Vineet

On Jul 20, 2018, at 2:07 PM, Vihang Karajgaonkar 
mailto:vih...@cloudera.com>> wrote:

Shouldn't these links include the metastore tar balls as well? I thought the 
plan was to release Metastore 3.1 along with Hive 3.1 based on the discussion 
we had on this list a few weeks ago. I am currently in process of verifying the 
standalone-metastore for branch-3.1

On Wed, Jul 18, 2018 at 6:15 PM, Vineet Garg 
mailto:vg...@hortonworks.com>> wrote:
Apache Hive 3.1.0 Release Candidate 0 is available here:

http://people.apache.org/~vgarg/apache-hive-3.1.0-rc-0

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1089/

Source tag: https://github.com/apache/hive/tree/release-3.1.0-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: [VOTE] Apache Hive 3.1.0 Release Candidate 0

[Vihang Karajgaonkar]

Shouldn't these links include the metastore tar balls as well? I thought
the plan was to release Metastore 3.1 along with Hive 3.1 based on the
discussion we had on this list a few weeks ago. I am currently in process
of verifying the standalone-metastore for branch-3.1

On Wed, Jul 18, 2018 at 6:15 PM, Vineet Garg  wrote:

> Apache Hive 3.1.0 Release Candidate 0 is available here:
>
> http://people.apache.org/~vgarg/apache-hive-3.1.0-rc-0
>
> Maven artifacts are available here:
>
> https://repository.apache.org/content/repositories/orgapachehive-1089/
>
> Source tag: https://github.com/apache/hive/tree/release-3.1.0-rc0
>
> Voting will conclude in 72 hours.
>
> Hive PMC Members: Please test and vote.
>
> Thanks.
>
>

Re: Review Request 67974: HIVE-20164

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67974/#review206291
---




ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Lines 1672 (patched)


Remove these comments?



ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Lines 1695 (patched)


please add curly braces



ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Lines 1701 (patched)


curly braces



ql/src/test/queries/clientpositive/murmur_hash_migration.q
Lines 36 (patched)


Does this test also need to query the inserted tables to show that things 
are working properly?


- Jason Dere


On July 19, 2018, 6:02 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67974/
> ---
> 
> (Updated July 19, 2018, 6:02 p.m.)
> 
> 
> Review request for hive, Gopal V and Jason Dere.
> 
> 
> Bugs: HIVE-20164
> https://issues.apache.org/jira/browse/HIVE-20164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Murmur Hash : Make sure CTAS and IAS use correct bucketing version
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties d08528f319 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 1b433c7498 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java bbce940c2e 
>   ql/src/test/queries/clientpositive/murmur_hash_migration.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/murmur_hash_migration.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67974/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Change patch naming convention

[Vihang Karajgaonkar]

Hi Daniel,

Yes, you are right. Looks like it doesn't work HIVE-1234.01-branch-3.patch
format. I think the reason it was done that way is because we use '-' in
the branch naming convention and its easier to parse the patch name to
determine the profile. I think it would be helpful to support both '.' and
'-' to delimit the branch names. The code which parses the patch name to
figure out the profile is here
https://github.com/apache/hive/blob/master/dev-support/jenkins-common.sh#L95

Feel free to create a JIRA (or use HIVE-20191) to provide a patch to
improve this. I would be happy to review it.

Thanks,
Vihang

On Tue, Jul 17, 2018 at 11:43 PM, Dániel Vörös 
wrote:

> Hi,
>
> Thanks Zoltan for starting this discussion. I ran into this a couple times
> lately.
>
> I've just opened https://issues.apache.org/jira/browse/HIVE-20191 to
> mitigate the
> effect of a wrong submission. Since I was backporting from master, I did't
> even
> notice that the patches were applied on master.
>
> I'd be in favor of changing the pattern, or allowing both '.' and '-'.
>
> @Vihang, from the Hive QE comments it seems that your first two patches
> were
> also applied on master instead of branch-3.
>
> Regards,
> Daniel
>
> On Tue, Jul 17, 2018 at 6:29 PM Vihang Karajgaonkar
>  wrote:
>
> > It doesn't matter if you use a dot v/s a dash to separate the patch
> version
> > with the profile name (same as branch name but its technically
> different).
> > See https://issues.apache.org/jira/browse/HIVE-18885 as an example
> where I
> > had submitted both the naming styles and precommit picked it up. Having
> > said that I find this naming convention confusing too. Does anyone know
> how
> > do the other projects handle branch specific patch names?
> >
> > On Tue, Jul 17, 2018 at 2:26 AM, Zoltan Haindrich  wrote:
> >
> > > Hello!
> > >
> > > I've seen a few people running into some problems because the current
> > > patch naming convention is a little bit odd:
> > >
> > > HIVE-[.][-].patch
> > >
> > > ...so if you would like to submit patch #02 to branch-2:
> > >
> > > A file named: HIVE-111.02-branch-2 patch should be uploaded (note that
> > > there is a mandatory dash after the patch number)
> > > I think this looks a bit odd also because most of the branch-es has a
> "-"
> > > in there name (branch-2/branch-3).
> > >
> > > Because most people are naturally uploading files with the following
> > > format, I would like to recommend to change the convention to:
> > > HIVE-[.][.].patch
> > >
> > > Examples:
> > > HIVE-111.feature-branch.patch
> > > HIVE-111.123.feature-branch.patch
> > > HIVE-111.123.feature-branch.patch
> > >
> > > cheers,
> > > Zoltan
> > >
> >
>

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/
---

(Updated July 20, 2018, 6:15 p.m.)


Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Fixed incorrect last reviewboard update.


Summary (updated)
-

HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
HMS instance


Bugs: HIVE-20194
https://issues.apache.org/jira/browse/HIVE-20194


Repository: hive-git


Description
---

HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
HMS instance


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47f819b60a6c7eb2acac9217724e67a976341093 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 92e280570f44cd56fe87c3123001b9263d28186f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 d91f737450efa9de6db3806401072645c96bbb7d 


Diff: https://reviews.apache.org/r/67954/diff/4/

Changes: https://reviews.apache.org/r/67954/diff/3-4/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 67954: HIVE-20194:

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/
---

(Updated July 20, 2018, 6:14 p.m.)


Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Updated comments


Bugs: HIVE-20194
https://issues.apache.org/jira/browse/HIVE-20194


Repository: hive-git


Description
---

HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
HMS instance


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47f819b60a6c7eb2acac9217724e67a976341093 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 92e280570f44cd56fe87c3123001b9263d28186f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 d91f737450efa9de6db3806401072645c96bbb7d 


Diff: https://reviews.apache.org/r/67954/diff/3/

Changes: https://reviews.apache.org/r/67954/diff/2-3/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-20218:
-

 Summary: make sure Statement.executeUpdate() returns number of 
rows affected
 Key: HIVE-20218
 URL: https://issues.apache.org/jira/browse/HIVE-20218
 Project: Hive
  Issue Type: Improvement
  Components: JDBC, Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


HiveStatement and HivePreparedStatement currently return 0 in all cases



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20217) bucket7.q test results should be sorted

[Peter Vary (JIRA)]

Peter Vary created HIVE-20217:
-

 Summary: bucket7.q test results should be sorted
 Key: HIVE-20217
 URL: https://issues.apache.org/jira/browse/HIVE-20217
 Project: Hive
  Issue Type: Test
  Components: Hive, Test
Affects Versions: 4.0.0
Reporter: Peter Vary
Assignee: Peter Vary


The test results should be sorted, so they will be the same on different configs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67954: HIVE-20194:

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/
---

(Updated July 20, 2018, 6:40 a.m.)


Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Simplified the signature of reflection method.


Summary (updated)
-

HIVE-20194:


Bugs: HIVE-20194
https://issues.apache.org/jira/browse/HIVE-20194


Repository: hive-git


Description
---

HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
HMS instance


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47f819b60a6c7eb2acac9217724e67a976341093 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 92e280570f44cd56fe87c3123001b9263d28186f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 d91f737450efa9de6db3806401072645c96bbb7d 


Diff: https://reviews.apache.org/r/67954/diff/2/

Changes: https://reviews.apache.org/r/67954/diff/1-2/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

[Alexander Kolbasov]

> On July 19, 2018, 12:25 a.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> > Lines 111-116 (patched)
> > 
> >
> > Do you think we can get these values from a config instead of hard 
> > coding?

Hmm, it is possible, I am not sure whether it is worth introducing a config 
variable fo this. This is a one-off case. Do you think it is better to make 
configurable?


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/#review206230
---


On July 18, 2018, 12:10 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67954/
> ---
> 
> (Updated July 18, 2018, 12:10 a.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-20194
> https://issues.apache.org/jira/browse/HIVE-20194
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
> HMS instance
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47f819b60a6c7eb2acac9217724e67a976341093 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  92e280570f44cd56fe87c3123001b9263d28186f 
>   
> standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  d91f737450efa9de6db3806401072645c96bbb7d 
> 
> 
> Diff: https://reviews.apache.org/r/67954/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>