[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-04-03 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424311#comment-16424311
 ] 

Alan Gates commented on HIVE-18755:
---

I've filed HIVE-19099 for this issue.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.final.patch, HIVE-18755.nothrift, 
> HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-04-02 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423031#comment-16423031
 ] 

Alan Gates commented on HIVE-18755:
---

Alright, I'll file a ticket and get this fixed.  Thanks for catching this.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.final.patch, HIVE-18755.nothrift, 
> HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-04-02 Thread Vineet Garg (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422929#comment-16422929
 ] 

Vineet Garg commented on HIVE-18755:


Workaround is to initSchema to 2.3.0 and then upgrade to 3.0.0

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.final.patch, HIVE-18755.nothrift, 
> HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-04-02 Thread Vineet Garg (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422914#comment-16422914
 ] 

Vineet Garg commented on HIVE-18755:


[~alangates] I guess 
{{metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql}} needs to be 
updated to create DBS table?

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.final.patch, HIVE-18755.nothrift, 
> HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-04-02 Thread Vineet Garg (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422910#comment-16422910
 ] 

Vineet Garg commented on HIVE-18755:


Looks like this has broken the internal derby initschema. I am unable to start 
hive cli. I get the following error:

{code:sql}
Exception in thread "main" java.lang.RuntimeException: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:651)
 at 
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:560)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
 at 
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:894)
 at 
org.apache.hadoop.hive.ql.session.SessionState.getAuthenticator(SessionState.java:1619)
 at 
org.apache.hadoop.hive.ql.session.SessionState.getUserFromAuthenticator(SessionState.java:1236)
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.setupSessionAcls(TezSessionState.java:552)
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:348)
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.beginOpen(TezSessionState.java:248)
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:648)
 ... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
 at 
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:926)
 at 
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:886)
 ... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
 at 
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:261)
 at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:413)
 at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:357)
 at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:337)
 at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:313)
 at 
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:922)
 ... 16 more
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
 at 
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
 at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:95)
 at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
 at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
 at 
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4062)
 at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4114)
 at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4094)
 at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4350)
 at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:273)
 at 
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:256)
 ... 21 more
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at 
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
 ... 30 more
Caused by: MetaException(message:Insert of object 
"org.apache.hadoop.hive.metastore.model.MDatabase@1e76afeb" using statement 
"INSERT INTO DBS 
(DB_ID,CTLG_NAME,"DESC",DB_LOCATION_URI,"NAME",OWNER_NAME,OWNER_TYPE) VALUES 
(?,?,?,?,?,?,?)" failed : 'CTLG_NAME' is not a column in table or VTI 
'APP.DBS'.)
 at

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-30 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421125#comment-16421125
 ] 

Alan Gates commented on HIVE-18755:
---

In a local run of the tests everything that passes in master passes on my 
changes.  The logs are large but I can post them if there's interest.  

Let's do this on Alexander's comments, since he's out for a couple of weeks.  
We'll take these as follow up items.  Any issues should be attached to 
HIVE-18685 so that they aren't lost.  I'll do the same with Thejas comment on 
the AuthorizationMetaStoreFilterHook and mark that as a blocker for 3.0.  I 
also need to file JIRAs for the TODO CAT comments as Thejas suggested 
previously.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-30 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421116#comment-16421116
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

I checked ObjectStore changes and put some comments there. I will not be able 
to continue review until April 11.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-30 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420929#comment-16420929
 ] 

Thejas M Nair commented on HIVE-18755:
--

[~akolb]
The github pull request is https://github.com/apache/hive/pull/320


> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-30 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420908#comment-16420908
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

[~alangates] Do you have either reviewboard or git pull request with the final 
proposed changes? What should we use to review the changes?

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-30 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420877#comment-16420877
 ] 

Thejas M Nair commented on HIVE-18755:
--

+1 pending tests

It looks like the metastore client getTableMeta is not getting filtered any 
more by the authorization hook. filterNames method was being used to filter 
that, but that has been removed.
There is a new filterHook.filterTableMetas method being used, but that is not 
implemented in AuthorizationMetaStoreFilterHook.  We can take that as follow up 
jira since this is a very large patch that can go out of sync again soon.



> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-27 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416454#comment-16416454
 ] 

Hive QA commented on HIVE-18755:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
18s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
16s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
34s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
41s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
20s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 20s{color} 
| {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} hcatalog/core: The patch generated 1 new + 10 
unchanged - 1 fixed = 11 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 20 new + 7 
unchanged - 18 fixed = 27 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
22s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 623 
unchanged - 12 fixed = 635 total (was 635) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
52s{color} | {color:red} ql: The patch generated 18 new + 1245 unchanged - 13 
fixed = 1263 total (was 1258) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
49s{color} | {color:red} standalone-metastore: The patch generated 956 new + 
2265 unchanged - 436 fixed = 3221 total (was 2701) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 154 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
4s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
8s{color} | {color:red} standalone-metastore generated 1 new + 52 unchanged - 9 
fixed = 53 total (was 61) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9875/dev-support/hive-personality.sh
 |
| git revision | master / f82eb8d |
| Default Java | 1.8.0_111 |
| mvninstall |

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-26 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414841#comment-16414841
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

[~alangates] Can you also update the pull request with your new changes? Thanks!

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-26 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414784#comment-16414784
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

[~alangates] Thanks for your replies. A few more comments.

# Regarding optional vs non-optional. Your reasoning makes sense, but we will 
have some calls where catalogs must be optional (for compat reasons). As a 
result, for some calls it is optional and for some it is required which may be 
confusing - some calls need to check that the value is present or not, some 
don't and this will complicate things, so making them uniformly optional may be 
just simpler. If you think that's not an issue it is fine as well.
# {quote}On the comments about getCatalogs(), I don't plan to add a request 
structure as I don't envision there ever being a structure above catalogs 
(since that's the way SQL defines it).  {quote}. Here is an example use case - 
your proposed use of catalogs to distinguish between Hive catalog and, say, 
Kafka catalog. On top of that we want to add "cluster" namespace which 
distinguish what luster accesses the data and all namespaces may deal with hive 
and kafka catalogs. Even if you think that this isn't a valid use case now, it 
may become useful soon, so wouldn't it be better to avoid running into the same 
issues soon afterwards?
# {quote}As to getCatalogs returning a list of strings, it matches the name of 
getDatabases, which returns a list of DB names. {quote} This doesn't make 
getDatabases very good API either, so is it better to fix newer ones or 
duplicate old bad ones?
# On auto-creation your reasoning makes sense.
# For Location - I see what you mean, can you a) document that and b) make 
location optional since there are likely to be catalogs for which location 
isn't anyhow useful.


> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-26 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414767#comment-16414767
 ] 

Alan Gates commented on HIVE-18755:
---

Uploaded new version of the patch that fixes a couple of the bugs I found, 
rebases against master, and addresses most of Thejas' and Alexander's comments. 
 I haven't taken the TODO CAT comments out yet, but I'll do that before I 
commit.  I'll put them in HIVE-18685 since that is the main Jira for the adding 
catalogs.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, 
> HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-26 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414149#comment-16414149
 ] 

Alan Gates commented on HIVE-18755:
---

I will change the various calls to take unique requests.  

I don't want to make the catalogs optional anywhere.  Going forward we want 
them to be part of all calls.  As the API grows (and hopefully as we build a 
v2) we want all new calls to include them.  

On the comments about getCatalogs(), I don't plan to add a request structure as 
I don't envision there ever being a structure above catalogs (since that's the 
way SQL defines it).  

As to getCatalogs returning a list of strings, it matches the name of 
getDatabases, which returns a list of DB names.  

On auto-creation of catalogs, I definitely do not want to do that.  When I get 
to the security section my intention is to make it so that only admin's can 
create catalogs.  How I'll enforce that on the metastore side I haven't figured 
out.  But I don't want it to be easy to create catalogs.

On catalog location, it needs a default location.  Today if you create a 
database without specifying a location it defaults to the warehouse root.  That 
won't work once we have catalogs because there could be two databases D (one in 
catalog C1 and one in catalog C2) both created without a location specified.  
Therefore a catalog has to have a location.  The 'hive' catalog (the default) 
has a location of the warehouse root to match the current behavior.  Once we 
have to have the location it makes sense to make it a field, since it will 
always be there.
{quote}Note that this removes use of ConcurrentMaps with non concurrent ones in 
SharedCache. But with the read/write locking that is already in place, that 
seems to be fine.
{quote}
It was not my intent to change anything in SharedCache other than make it work 
with catalogs.  Any change from ConcurrentMap to Map was unintentional.  I 
agree Daniel and Vaibhav should take a look to make sure my changes there make 
sense.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-26 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413437#comment-16413437
 ] 

Thejas M Nair commented on HIVE-18755:
--

I have added some comments in github pull request.


> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-25 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413236#comment-16413236
 ] 

Thejas M Nair commented on HIVE-18755:
--

[~vgumashta] [~daijy]
Note that this removes use of ConcurrentMaps with non concurrent ones in 
SharedCache. But with the read/write locking that is already in place, that 
seems to be fine.


> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412282#comment-16412282
 ] 

Thejas M Nair commented on HIVE-18755:
--

On the lines of what Alexander said, I think it makes sense to create a 
GetCatalogReq object and DropCatalogReq objects for use in get_catalog and 
drop_catalog methods respectively. You might have arguments that are specific 
to drop_catalog in future  (eg "cascade=true").

Similarly even for create_catalog, having a CreateCatalogReq object would help 
to keep it future proof.  Now thinking on those lines, I think its simpler are 
more consistent to use *Req and *Response objects as far as possible.





> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412133#comment-16412133
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

Thanks for the clarification - it isn't immediately clear which structures are 
old and which are new. Any reason not be consistent and make catalog always 
optional?

Your reasoning for CatalogName makes sense but what do you think about having a 
request structure instead?

Some more questions.

# Your current API assumes that catalogs must be created before they can be 
used. Is there any value in auto-creating them on first use or this is too 
complicated?
# I don't get the argument about Location - what does it mean 'place where data 
in the catalog will be stored?' Different data may be stored in different 
places. Even in the case of Hive every Database has its own location so what 
does it mean to have a catalog location? Is it some kind of default for 
Database locations when they are not specified? Why not just have Parameters 
instead (which may be way more useful) and if someone wants to store Location 
parameter it is just another parameter. Ot you envision HMS actually 
interpreting this Location field in some form?
# create_catalog() has Catalog as the only argument. What of later you want to 
change this to take more arguments? Wouldn't it be better to have 
CreateCatalogRequest?
# Why get_catalogs returns list of strings rather then list of catalogs? At the 
minimum the name is misleading. Also, would it make sense to add Request 
structure there by the time you'll want to add a namespace that all catalogs 
belong to?
# # drop_catalog - should it get request structure as an arg? What happens with 
all things inside the catalog when catalog is dropped? What is the general 
semantics?

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411634#comment-16411634
 ] 

Alan Gates commented on HIVE-18755:
---

Responses to [~akolb]'s comments:

I don't think I broke wire compatibility anywhere.  I made catalog required 
only in newer structs that have never been in a release (like 
DefaultContraintRequest) and optional in anything that has been in a release.

I can add comments to the thrift code.

CatalogName is an attempt to avoid the problem we have with Databases and 
Tables and such where we are putting the string values directly in the method 
calls, making it hard to add arguments to the method later.  It makes 
getCatalog and dropCatalog future proof.

Catalog location is the same thing as dblocation, the place where data in the 
catalog will be stored.  It's not optional because we have to put the data 
somewhere.  I could have inferred it from the catalog name, but it doesn't seem 
unreasonable to require an administrator to designate a location for a catalog. 
 

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411627#comment-16411627
 ] 

Hive QA commented on HIVE-18755:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915734/HIVE-18755.2.patch

{color:green}SUCCESS:{color} +1 due to 51 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 463 failed/errored test(s), 9606 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=210)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10] 
(batchId=251)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_database]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like] 
(batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_nonpart]
 (batchId=254)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
 (batchId=254)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=29)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411584#comment-16411584
 ] 

Hive QA commented on HIVE-18755:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
21s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
34s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
44s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
19s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 19s{color} 
| {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} hcatalog/core: The patch generated 1 new + 10 
unchanged - 1 fixed = 11 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} itests/hcatalog-unit: The patch generated 18 new + 7 
unchanged - 16 fixed = 25 total (was 23) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
22s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 623 
unchanged - 12 fixed = 635 total (was 635) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
51s{color} | {color:red} ql: The patch generated 16 new + 1226 unchanged - 8 
fixed = 1242 total (was 1234) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
46s{color} | {color:red} standalone-metastore: The patch generated 930 new + 
2244 unchanged - 420 fixed = 3174 total (was 2664) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 138 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
3s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
2s{color} | {color:red} standalone-metastore generated 1 new + 52 unchanged - 9 
fixed = 53 total (was 61) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9783/dev-support/hive-personality.sh
 |
| git revision | master / cc7377c |
| Default Java | 1.8.0_111 |
| mvninstall |

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-23 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410925#comment-16410925
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

Looking at thrift changes first. A few comments there:

# There are many cases where the catalog is not added as the last field but 
elsewhere in the existing structure and some places where it is added as 
required field. I think this breaks on the wire compatibility for Thrift.
# Please document *all* additions to the thrift file. It is an API spec and not 
having comments there is completely unacceptable.

The spec for catalog says:

{code}
struct Catalog {
  1: string name,
  2: optional string description,
  3: string locationUri
}

struct CatalogName {
  1: string name
}
{code}

What is CatalogName and why it is a separate struct?
What is Location for catalog and why it is not optional?

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-22 Thread Peter Vary (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409298#comment-16409298
 ] 

Peter Vary commented on HIVE-18755:
---

Thanks Alan, that makes perfect sense. I see it now!

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-21 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408812#comment-16408812
 ] 

Alan Gates commented on HIVE-18755:
---

Peter, I think I understood your intent from the first question, which I 
believe was: "why not always handle the missing catalog on the server side?"  
To which my answer is, I am trying to always handle the missing catalog on the 
server side, so that it is compatible with old clients.  But I am also handling 
the missing catalog on the client side (like my change in DDLTask.java) so that 
if the metastore server is set up to talk to one catalog but users want to 
configure their clients to talk to another by default without changing every 
one of their client calls to use the new catalog aware method, they can.  So 
for example, say someone has a Hive CLI client they are using and they want to 
default to 'mycatalog' instead of whatever their metastore server is set to.  
They can then set that value in their config file and it will still work, even 
though I haven't changed every call to getTable etc inside Hive.  

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18755.nothrift, HIVE-18755.patch
>
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-20 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407219#comment-16407219
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

[~alangates] I am trying to apply the patch but it doesn't apply cleanly to 
master. What is the tip commit for which your patch was applied? 

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-20 Thread Peter Vary (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406760#comment-16406760
 ] 

Peter Vary commented on HIVE-18755:
---

Uhh.. :( Communication error on my side. Reading back what I have wrote, I do 
not know how I ended up with this sentence :(

I was referring this kind of changes, when we are using the default value not 
in the HiveMetaStore code:
{code:java|title=ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java}
private int createView(Hive db, CreateViewDesc crtView) throws HiveException {
[..]
new CreationMetadata(MetaStoreUtils.getDefaultCatalog(conf), tbl.getDbName()
[..]
}
{code}
Maybe doing this defaulting when using the CreationMetadata values would be 
better. Why did you decided against it?

Thanks,
Peter


> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-20 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406694#comment-16406694
 ] 

Alan Gates commented on HIVE-18755:
---

{quote}Is it possible to keep the default catalog a strictly server side 
configuration
{quote}
Possible, but not desirable.  I want to enable the case where the client wants 
to use a different catalog then the server is configured for.  Consider the 
case of a user who has one metastore set up, but wants some clients to default 
to one catalog and one to another.  By allowing the client to be configured to 
a default catalog we can support this.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-20 Thread Peter Vary (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406158#comment-16406158
 ] 

Peter Vary commented on HIVE-18755:
---

Just a quick question: Is it possible to keep the default catalog a strictly 
server side configuration, or we do have to leak it to client side as well?

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-20 Thread Peter Vary (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406084#comment-16406084
 ] 

Peter Vary commented on HIVE-18755:
---

Thanks [~alangates], good to know that our work with [~kuczoram], and [~szita] 
already paid some dividends :D :D :D

This is a huge patch. I will try to find time to review.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-15 Thread Alexander Kolbasov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401283#comment-16401283
 ] 

Alexander Kolbasov commented on HIVE-18755:
---

[~alangates] Can you post a patch against master? Reviewing it from github 
isn't the best option.

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-15 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401239#comment-16401239
 ] 

Alan Gates commented on HIVE-18755:
---

The PR is not yet ready for merging, I'm just posting it as preliminary work, 
since the patch is huge.

So, let me begin by apologizing for the enormity of this patch.  I didn't see a 
good way of breaking it up without leaving lots of half baked stuff in there, 
so I did almost all of the metastore work in one pass.

Second, let me say my sincerest thanks to [~pvary] and others who added all the 
client tests.  They were invaluable in helping me test the new code.

A comment where I took a different path than in the doc posted at HIVE-18685.  
Rather than hard wiring all the old calls to the 'hive' catalog I added a 
notion of default catalog in the config file.  This defaults to 'hive'.  This 
allows an admin to set up a metastore server to talk to a non-'hive' catalog 
while still using older clients.  This seems like a useful feature.

While I have included a lot of testing in this patch there is still a fair 
amount to add.  Things I have tested:
 * new catalog calls
 * existing calls without catalog set
 * direct SQL access path for reading data out of the database.

Things I haven't tested yet but plan to before committing the patch:
 * an old client talking to the server; I want to make sure this works even 
though the old client will have no notion of catalogs
 * configuring the default catalog to be something other than 'hive' and making 
sure a client still works
 * the JDO access path for reading data out of the database

> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs

2018-03-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401217#comment-16401217
 ] 

ASF GitHub Bot commented on HIVE-18755:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/320

HIVE-18755 Modifications to metastore for catalog



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive18755

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/320.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #320


commit 697e0428b4046b0c36d413a74fc46c2cf28d6e67
Author: Alan Gates 
Date:   2018-03-15T22:51:06Z

HIVE-18755 Modifications to metastore for catalog




> Modifications to the metastore for catalogs
> ---
>
> Key: HIVE-18755
> URL: https://issues.apache.org/jira/browse/HIVE-18755
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> Step 1 of adding catalogs is to add support in the metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)