[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396401#comment-16396401
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


patch merged to standalone-metastore branch. Thanks [~alangates] for the review 
and [~owen.omalley] for the guidance.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394361#comment-16394361
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Fixed TestAvroObjectInspectorGenerator. Rest of the failures are unrelated.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394170#comment-16394170
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913856/HIVE-17580.093-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13214 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
 (batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] 
(batchId=251)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=222)
org.apache.hadoop.hive.serde2.avro.TestAvroObjectInspectorGenerator.canHandleArrays
 (batchId=300)
org.apache.hadoop.hive.serde2.avro.TestAvroObjectInspectorGenerator.canHandleMapsWithPrimitiveValueTypes
 (batchId=300)
org.apache.hadoop.hive.serde2.avro.TestAvroObjectInspectorGenerator.canHandleRecords
 (batchId=300)
org.apache.hadoop.hive.serde2.avro.TestAvroObjectInspectorGenerator.convertsMapsWithNullablePrimitiveTypes
 (batchId=300)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9589/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9589/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9589/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12913856 - PreCommit-HIVE-Build

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is 

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393906#comment-16393906
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Thanks for the review [~alangates] Attached the latest patch from the PR to run 
the precommit tests to make sure there is nothing fundamentally broken. If 
there are no unexpected failures, I will merge the PR into standalone-metastore 
on Monday.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392082#comment-16392082
 ] 

Alan Gates commented on HIVE-17580:
---

+1 for the latest PR.  Thank Vihang, this was a lot of work.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392022#comment-16392022
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


I update the PR to include the above proposed change in case anyone wants to 
take a look.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388946#comment-16388946
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Thanks [~owen.omalley] for explaining. In that case I think it is better to 
break compatibility on TypeInfo.getCategory() method signature. There hasn't 
been any released version of Hive which has TypeInfo annotated as public API so 
this is as good a time to fix this as any. IMHO Category enum should not be in 
ObjectInspector in the first place. Instead of bending backwards to accommodate 
a design choice which doesn't make sense I think we should fix it. I propose we 
should introduce a new enum called {{MetastoreTypeCategory}} in metastore and 
change the signature of {{getCategory()}} method in {{TypeInfo}} to return 
{{MetastoreTypeCategory}} instead of {{ObjectInspector.Category}}. 
{{ObjectInspector.Category}} would be modified so that there are helper methods 
to map one to another. The values of the enums will also remain the same, so as 
long as someone is using this enum for switch statements it would just work 
without any changes. However, if some project outside hive is using it for 
equality checks, it will not work. Eg:

{{typeInfo.getCategory() == ObjectInspector.Category.PRIMITIVE}} will not work. 
Instead users should use the utility methods to convert them and compare. So in 
this case they could do a {{typeInfo.getCategory() == 
ObjectInspector.Category.PRIMITIVE.toMetastoreTypeCategory()}} to get the 
previous behavior.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388658#comment-16388658
 ] 

Owen O'Malley commented on HIVE-17580:
--

{quote}I don't see a particular logically grouping of classes in storage-api 
(for example, HiveDecimal is in storage-api, but the other types are in serde). 
I think in the longer run we would need to reorganizing this in more consistent 
modules anyways.
{quote}
I've tried, but not always succeeded. HiveDecimal is there, because it is used 
by DecimalColumnVector. The other types aren't used by the other ColumnVectors. 
It would be better if DecimalColumnVector did not use HiveDecimal, but that 
would take a lot of work to fix.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388655#comment-16388655
 ] 

Owen O'Malley commented on HIVE-17580:
--

{quote}The other option suggested of moving ObjectInspector to 
standalone-metastore sounds weird to me since it has got nothing to do with 
metastore. ** 
{quote}
It has even less to do with storage-api, which is precisely why I don't want it 
there. Even worse, it moves really high up the release tree:
 # storage-api
 # ORC
 # metastore
 # hive

Moving things into storage-api that don't need to be there is a big cost. 
Moving it to metastore that actually does need the enum makes at least some 
sense. Nothing in storage-api or ORC need or want that enum.

 

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387354#comment-16387354
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Combining the enums is not going to help solve the problem, since metastore 
needs to get compile time access to ObjectInspector.TypeCategory unless we are 
okay to break compatibility by modifying the signature of getCategory method in 
TypeInfo. TypeInfo was annotated as public in HIVE-17157 so technically there 
hasn't been any released version of Hive which has it as a public API. So I 
think we are allowed to break compatibility with that respect. What do you 
think?

The other option suggested of moving ObjectInspector to standalone-metastore 
sounds weird to me since it has got nothing to do with metastore. Moving 
ObjectInspector to storage-api is not ideal but atleast is lesser of the two 
evils. It could be argued that it more a storage-api than a metastore api since 
it is related to how to interpret the serialized and deserialized data. I don't 
see a particular logically grouping of classes in storage-api (for example, 
HiveDecimal is in storage-api, but the other types are in serde). I think in 
the longer run we would need to reorganizing this in more consistent modules 
anyways.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386564#comment-16386564
 ] 

Owen O'Malley commented on HIVE-17580:
--

The problem with putting ObjectInspector into storage-api is that 
ObjectInspector by itself doesn't do anything. You need the cloud of stuff 
around ObjectInspector to do anything. It is also ill fitting because 
storage-api is the *vectorized* api. It by design does not include the 
ObjectInspector and the associated slow legacy path for Hive.

Maybe a simpler fix is to move ObjectInspector into standalone metastore and 
make serde depend on it. That would at least not pull ObjectInspect into the 
storage-api and only put it where need is.

You could even combine the two with:
{{public enum MetastoreTypeCategory {...};}}

and have ObjectInspector.TypeCategory with:
{{public MetastoreTypeCategory toMetastore();}}


> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382432#comment-16382432
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


[~owen.omalley] [~alangates] any thoughts on my comment above?

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381165#comment-16381165
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


on a side note, took a look at the test failures on the latest patch. The 
{{TestJdbcWithMiniLlap}} look environment related. These tests are working for 
me locally. TestNegativeCliDriver.testCliDriver[avro_decimal] would require a 
q.out update. TestHCatLoaderComplexSchema works for me locally. 
TestSchemaToTypeInfo needs to change to to fix since the expected exception is 
different. org.apache.hadoop.hive.metastore.client test failures are unrelated 
to this change.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381124#comment-16381124
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Thanks [~owen.omalley] for taking a look.

bq. I'm not happy about ObjectInspector moving into storage-api.

Can you please give reasons why you think its not a good idea to move just the 
interface to storage-api?

bq. If you need the Category enum, I'd suggest you duplicate it and extend the 
ObjectInspector.Category with a mapping to the storage-api Category.

Just to confirm I understand what you are suggesting above. Lets say we create 
a duplicate Category enum called {{MetastoreTypeCategory}} and define it in 
metastore. We can add a methods in ObjectInspector.Category to map one to 
another. How does that help with the {{getCategory}} method in TypeInfo since 
it returns ObjectInspector.Category? It will still not compile. Are you also 
suggesting modify that method to return {{MetastoreTypeCategory}}? Not only 
that would be backwards incompatible for anyone outside hive using TypeInfo, it 
would also need modification all over the code (eg. If I find usages of this 
method in the code I see 221 usages across multiple modules) Its doable but is 
it worth it?

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381021#comment-16381021
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912459/HIVE-17580.092-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 13193 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin6]
 (batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
 (batchId=180)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_decimal] 
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] 
(batchId=251)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoTblName[Embedded]
 (batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
 (batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
 (batchId=206)
org.apache.hadoop.hive.serde2.avro.TestSchemaToTypeInfo.testDisallowRecursiveSchema
 (batchId=300)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[2]
 (batchId=193)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testDataTypes (batchId=236)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testEscapedStrings (batchId=236)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testKillQuery (batchId=236)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=236)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testNonAsciiStrings (batchId=236)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9408/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9408/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9408/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12912459 - PreCommit-HIVE-Build

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> 

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380792#comment-16380792
 ] 

Owen O'Malley commented on HIVE-17580:
--

I'm not happy about ObjectInspector moving into storage-api.

If you need the Category enum, I'd suggest you duplicate it and extend the 
ObjectInspector.Category with a mapping to the storage-api Category.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380769#comment-16380769
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912387/HIVE-17580.091-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13214 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
 (batchId=180)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_decimal] 
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] 
(batchId=251)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullFunctionNameInNew[Remote]
 (batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
 (batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
 (batchId=206)
org.apache.hadoop.hive.serde2.avro.TestSchemaToTypeInfo.testDisallowRecursiveSchema
 (batchId=300)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=232)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=232)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill 
(batchId=236)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9406/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9406/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9406/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12912387 - PreCommit-HIVE-Build

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to 

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380695#comment-16380695
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


resubmitted patch to trigger precommit. 092 is the latest patch. #9​40​8 
precommit scheduled.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379860#comment-16379860
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


latest patch restores {{Category}} enum back into {{ObjectInspector}} since 
apparently projects outside Hive also use it directly and moving it out of 
ObjectInspector will break stuff. This patch moves the {{ObjectInspector}} 
interface to storage-api instead of just moving Category out of it. This way 
compatibility is maintained and metastore can still access it without depending 
on hive-serde.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.091-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376509#comment-16376509
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


I tried to run TestTablesCreateDropAlterTruncate locally and it seems to be 
working fine. Is it 
{code}
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running 
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate
[INFO] Tests run: 114, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.29 
s - in org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 114, Failures: 0, Errors: 0, Skipped: 0
{code}

Same with TestTablesList
{code}
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hive.metastore.client.TestTablesList
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.267 
s - in org.apache.hadoop.hive.metastore.client.TestTablesList
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
{code}

org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_decimal] 
(batchId=95) is failing with a different error message now because metastore 
validation does not do a strict range check for decimal precision and scale as 
designed. The execution engine (hive) does the strict range check [1,38] and 
since the precision is evaluated as 0 it fails. I don't fully understand the 
test case though. I will update the q.out after checking with [~aihuaxu] who 
wrote that test.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376197#comment-16376197
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12911962/HIVE-17580.09-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13214 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
 (batchId=180)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_decimal] 
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] 
(batchId=251)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
 (batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
 (batchId=206)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing 
(batchId=288)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=236)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9362/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9362/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9362/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12911962 - PreCommit-HIVE-Build

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent 

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376164#comment-16376164
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


added {{mavenTestArgs = -Dtest.groups=""}} to standalone-metastore branch 
profile to run the checkin tests. Reattached the patch again

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375930#comment-16375930
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12911946/HIVE-17580.09-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 11487 tests 
executed
*Failed tests:*
{noformat}
TestAddAlterDropIndexes - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed 
out) (batchId=206)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestDatabases - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestFunctions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetListIndexes - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely 
timed out) (batchId=209)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely 
timed out) (batchId=215)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMetaStoreInitListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestMetastoreSchemaTool - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestObjectStoreInitRetry - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestObjectStoreSchemaMethods - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestRawStoreProxy - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestRemoteHiveMetaStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=210)
TestRemoteHiveMetaStoreIpAddress - did not produce a TEST-*.xml file (likely 
timed out) (batchId=204)
TestRemoteUGIHiveMetaStoreIpAddress - did not produce a TEST-*.xml file (likely 
timed out) (batchId=213)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely 
timed out) (batchId=215)
TestRetryingHMSHandler - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestSchemaToolForMetastore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=209)
TestSetUGIOnBothClientServer - did not produce a TEST-*.xml file (likely timed 
out) (batchId=207)
TestSetUGIOnOnlyClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=205)
TestSetUGIOnOnlyServer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=214)
TestTablesCreateDropAlterTruncate - did not produce a TEST-*.xml file (likely 
timed out) (batchId=206)
TestTablesGetExists - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestTablesList - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=164)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375907#comment-16375907
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Most of these failures are because ColumnType.java is missing timestamp with 
local time zone type. Added that to ColumnType.java. I think we can get rid of 
ColumnType.java from standalone-metastore since hive-serde now depends on 
standalone-metastore, we can move serde_constants.thrift to 
standalone-metastore. This will get rid of the duplicate constants in 
ColumnType.java and there would be less chance of us tripping over due to such 
mismatches between the two.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375663#comment-16375663
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12911877/HIVE-17580.08-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 82 failed/errored test(s), 11487 tests 
executed
*Failed tests:*
{noformat}
TestAddAlterDropIndexes - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed 
out) (batchId=206)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestDatabases - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestFunctions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetListIndexes - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely 
timed out) (batchId=209)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely 
timed out) (batchId=215)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMetaStoreInitListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestMetastoreSchemaTool - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestObjectStoreInitRetry - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestObjectStoreSchemaMethods - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestRawStoreProxy - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestRemoteHiveMetaStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=210)
TestRemoteHiveMetaStoreIpAddress - did not produce a TEST-*.xml file (likely 
timed out) (batchId=204)
TestRemoteUGIHiveMetaStoreIpAddress - did not produce a TEST-*.xml file (likely 
timed out) (batchId=213)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely 
timed out) (batchId=215)
TestRetryingHMSHandler - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestSchemaToolForMetastore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=209)
TestSetUGIOnBothClientServer - did not produce a TEST-*.xml file (likely timed 
out) (batchId=207)
TestSetUGIOnOnlyClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=205)
TestSetUGIOnOnlyServer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=214)
TestTablesCreateDropAlterTruncate - did not produce a TEST-*.xml file (likely 
timed out) (batchId=206)
TestTablesGetExists - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
TestTablesList - did not produce a TEST-*.xml file (likely timed out) 
(batchId=206)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic1] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic3] 
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[localtimezone] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz] (batchId=64)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375360#comment-16375360
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Posting the pre-commit result here. 294 tests failed
{noformat}
Test Result (294 failures / +254)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_char]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[typechangetest]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_varchar]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[varchar_union1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_udf2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_types_vectorization]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join]
Show all failed tests >>>
{noformat}

Updated the patch which fixes the error.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374823#comment-16374823
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


pre-commit didn't trigger for some reason. Reattaching again the patch again.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373700#comment-16373700
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Hi [~alangates] the latest patch attached here moves the TypeInfos (with much 
less disruption than previously thought) to standalone-metastore like we 
discussed last time. This would remove a lot of duplicate code from the 
previous patch and also I think is more sensible from design point of view. The 
only class which caused lot of changes to other files was {{Category}} enum 
from ObjectInspector. I did not understand why it was defined in 
ObjectInspector in the first place since it looks more related to TypeInfo than 
ObjectInspector to me. Rest of the changes are not too complex. I have tried to 
explain the reasoning behind some of weird changes in code via code comments 
and above. Also, added a unit test in metastore to test avro schemas and ran 
the existing q tests for avro before submitting the patch. Can you please take 
a look?

cc [~ashutoshc] [~sershe] [~owen.omalley] [~akolb]

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373685#comment-16373685
 ] 

ASF GitHub Bot commented on HIVE-17580:
---

GitHub user vihangk1 opened a pull request:

https://github.com/apache/hive/pull/310

HIVE-17580 : Remove dependency of get_fields_with_environment_context API 
to serde

This version of patch moves TypeInfo and its sub-classes to 
standalone-metastore. The motivation of doing this is that metastore needs the 
TypeInfo like classes to store the metadata about types. This is implemented by 
TypeInfos in Hive. Metastore needs this information because table like avro can 
define schema externally using url to a file containing schema or a string 
value of the schema added as a table property. In such cases metastore need to 
parse this information and convert them into FieldSchema. Before this patch 
this String->FieldSchema conversion was done using SerDes using the 
ObjectInspectors and the typeInfos from them. This patch bypasses a lot of that 
to remove the dependency to the SerDes such that it converts the String -> 
TypeInfo -> FieldSchema.

In order to achieve this and also for reducing duplicate code and a cleaner 
design, this patch moves TypeInfo and its subclasses (ListTypeInfo, 
MapTypeInfo, StructTypeInfo, UnionTypeInfo), TypeInfoParser to standalone 
metastore. In case of PrimitiveTypeInfo, Hive code has added lot more than just 
type metadata in PrimitiveTypeInfo. Specifically, PrimitiveTypeEntry, 
PrimitiveCategory is type implementation detail which cannot be moved to 
standalone-metastore. Not to mention bring in PrimitiveTypeEntry bring in a 
whole lot of dependent code with it. To workaround this issue, a new class 
called MetastorePrimitiveTypeInfo is introduced in standalone-metastore. This 
class contains only the information which is needed by metastore from 
PrimitiveTypeInfo and PrimitiveTypeInfo extends MetastorePrimitiveTypeInfo. 
This way we reduce the scope of changes greatly. PrimitiveTypeInfo now contains 
implementation details of Hive's primitive types. Moving TypeInfo to 
standalone-metastore also needs the Category enum which unfortunately was 
defined in ObjectInspector. This is no way around this and this patch had to 
move Category to TypeInfo from ObjectInspector. Most of the file changes are 
due to this move.

Moving TypeInfoFactory was also very disruptive and hence an interface 
called ITypeInfoFactory is created in metastore and both metastore and hive 
implement this interface. The Avro storage schema reader now can use the 
TypeInfoToSchema and SchemaToTypeInfo util classes (also moved to metastore) 
using the ITypeInfoFactory interface.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580v4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #310


commit 2bdf6e18132f99f8998eceed8af0b77865fd85d4
Author: Vihang Karajgaonkar 
Date:   2018-02-22T21:10:03Z

Moved TypeInfo to standalone-metastore

commit 756a394280d0a940b7dbcca05805a62978c4d8b2
Author: Vihang Karajgaonkar 
Date:   2018-02-22T22:35:50Z

Introduce Avro storage schema reader




> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335179#comment-16335179
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Added the fix for the failing tests. Updated the PR with the fix as well.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334582#comment-16334582
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Test failures look related. I will take a look.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334052#comment-16334052
 ] 

Hive QA commented on HIVE-17580:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907060/HIVE-17580.04-standalone-metastore.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 11598 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column2] 
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column3] 
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column] 
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_change_schema] 
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_charvarchar] 
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_comments] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_compression_enabled]
 (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_compression_enabled_native]
 (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_date] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal] 
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_native] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_old] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_deserialize_map_null]
 (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_evolved_schemas] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_joins] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_joins_native] 
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_native] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_nullable_fields] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_nullable_union] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned_native] 
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_sanity_test] 
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
 (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_literal] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_timestamp] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_type_evolution] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] 
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrotblsjoin] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compustat_avro] 
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_serde] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_avro_partition_uniontype]
 (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_null] 
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_add_column_extschema]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_non_nullable_union]
 (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_compression_enabled_native]
 (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_decimal_native]
 (batchId=118)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333955#comment-16333955
 ] 

ASF GitHub Bot commented on HIVE-17580:
---

GitHub user vihangk1 opened a pull request:

https://github.com/apache/hive/pull/294

HIVE-17580 Remove dependency of get_fields_with_environment_context API to 
serde

This is an alternative approach to the solve the dependencies with serdes 
for get_fields HMS API. The earlier attempt for HIVE-17580 was very disruptive 
since it attempted to move TypeInfo, and various Type implementations to 
storage-api and also created another module called serde-api.

This patch is a lot more cleaner and less disruptive. Instead of moving 
TypeInfo, it creates similar classes in standalone-metastore. The PR is broken 
into multiple commits with descriptive commit messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/294.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #294


commit 708443af3f6356ab73133e271cf00e3418ced8ef
Author: Vihang Karajgaonkar 
Date:   2018-01-21T23:54:04Z

Added MetastoreTypeInfo similar to TypeInfo

This patch adds classes similar to TypeInfo called MetastoreTypeInfo in 
standalone-metastore.
Ideally, we should move TypeInfo to standalone-metastore since they store 
the information
about types. However, moving TypeInfo to standalone-metastore is 
non-trivial effort primarily
because of the below reasons:

1. TypeInfo is annotated as Public API.
This means we can only alter/move these classes in a compatible way.
2. Directly moving these classes is not straight-forward because TypeInfo 
uses PrimitiveEntry
class which internally maps the TypeInfo to Type implementations. Ideally 
metastore should
not use Type implementation which makes it harder to move the TypeInfo 
directly.

However, if we are ready to break compatibility, then TypeInfo broken such 
that it doesn't
use PrimitiveEntry directly. In such a world TypeInfo will store just what 
it needs to store.
Metadata of Types i.e the type category, its qualified name, whether its a 
parameterized type
or not and if yes, how do we validate the parameters.

I am assuming that breaking TypeInfo is a no-go and hence I am copying the 
relevant code
from TypeInfo to Metastore and calling it MetastoreTypeInfo. 
MetastoreTypeInfo and its sub-classes
are used by TypeInfoParser (also copied) to parse the column type strings 
into TypeInfos.

commit 6ec0efa59408c355cfa9aec7fd9dd59d3545aff2
Author: Vihang Karajgaonkar 
Date:   2018-01-03T19:45:32Z

Add avro storeage schema reader

This commit adds a AvroStorageSchemaReader which reads the Avro schema 
files both for external schema and regular avro tables.
Most of the util methods are in AvroSchemaUtils class which has methods 
copied from AvroSerDeUtils. Some of the needed classes like
SchemaResolutionProblem, InstanceCache, SchemaToTypeInfo, TypeInfoToSchema 
are also copied from Hive. The constants defined
in AvroSerde are copied in AvroSerdeConstants. The class 
AvroFieldSchemaGenerator converts the AvroSchema into List of
FieldSchema which is returned by the AvroStorageSchemaReader

Avro schema reader uses  MetastoreTypeInfo and MetastoreTypeInfoParser 
introduced earlier

commit b0f6d1df1ddb627e0f3c1cff3a164c9397337be0
Author: Vihang Karajgaonkar 
Date:   2018-01-04T01:02:40Z

Introduce default storage schema reader

This change introduces a default storage schema reader which copies the 
common code from serdes
initialization method and uses it to parse the column name, type and 
comments from the table
properties. For custom storage schema reades like Avro we will have to add 
more schema readers
as and when required

commit 5ae977a0bf3fd54389671bed86322d3d4652bc20
Author: Vihang Karajgaonkar 
Date:   2018-01-04T19:18:03Z

Integrates the avro schema reader into the DefaultStorageaSchemaReader

commit 2074b16e12c1bdc7ef3781f50e01ab4dd4c71890
Author: Vihang Karajgaonkar 
Date:   2018-01-05T02:38:28Z

Added a test for getFields method in standalone-metastore

commit 4159b5ee9852b41a64489274040e79dbddad54f1
Author: Vihang Karajgaonkar 
Date:   2018-01-22T07:16:13Z

HIVE-18508 : Port schema changes from HIVE-14498 to standalone-metastore




> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: 

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323292#comment-16323292
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


The reason why most of the tests are failing is because its not able to create 
tables. When I debugged it looks like its because the static initialization 
block of {{PrimitiveObjectInspectorUtils}} is not getting called which doesn't 
populate  the tables using {{registerType}} method calls. I will investigate 
how can I fix this.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318918#comment-16318918
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


[~alangates] I think its failing because its not able to find the profile 
corresponding to the branch standalone-metastore.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
> Attachments: HIVE-17580-standalone-metastore.001.patch, 
> HIVE-17580-standalone-metastore.002.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318831#comment-16318831
 ] 

Alan Gates commented on HIVE-17580:
---

I suspect HiveQA is getting tripped up by 'standalone-metastore' being both a 
branch and a directory.  Certain git commands don't work as expected due to 
this double use.  It was seriously short sighted of me to name both the 
directory and the branch with the same name.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
> Attachments: HIVE-17580-standalone-metastore.001.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318824#comment-16318824
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


HiveQA does not seem to like patches on standalone-metastore branch. Copying 
the console output here:

{noformat}
 local 
'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
+ java -cp 
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
 org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
 --password '[***]' --testHandle PreCommit-HIVE-Build-8511 --endpoint 
http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint 
http://104.198.109.242/logs/ --profile standalone-metastore.001-mr2 --patch 
https://issues.apache.org/jira/secure/attachment/12905154/HIVE-17580-standalone-metastore.001.patch
 --jira HIVE-17580


Logs are located: http://104.198.109.242/logs/PreCommit-HIVE-Build-8511


Exception in thread "main" java.lang.RuntimeException: Status 
[name=ILLEGAL_ARGUMENT, message=Profile standalone-metastore.001-mr2 not found 
in directory /usr/local/hiveptest/profiles]
at org.apache.hive.ptest.api.Status.assertOKOrFailed(Status.java:69)
at 
org.apache.hive.ptest.api.client.PTestClient.testTailLog(PTestClient.java:178)
at 
org.apache.hive.ptest.api.client.PTestClient.testStart(PTestClient.java:135)
at 
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:320)
+ ret=1
+ unpack_test_results
+ '[' -z /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build 
']'
+ cd 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
+ [[ -f test-results.tar.gz ]]
+ exit 1
+ rm -f /tmp/tmp.veDcfsGbID
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/test-results/TEST-1-TestCliDriver-udf_upper.q-ctas_date.q-schema_evol_orc_acidvec_table_update.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
 is 2 days 0 hr old

[description-setter] Description set: HIVE-17580  /   
standalone-metastore.001-mr2
Finished: FAILURE
{noformat}

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
> Attachments: HIVE-17580-standalone-metastore.001.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314314#comment-16314314
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


[~alangates] [~owen.omalley] and [~sershe] Can you please take a look? I can 
create multiple sub-tasks to break this request into smaller bits :) Thanks a 
lot!

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314309#comment-16314309
 ] 

ASF GitHub Bot commented on HIVE-17580:
---

GitHub user vihangk1 opened a pull request:

https://github.com/apache/hive/pull/287

HIVE-17580 : Remove standalone-metastore's dependency with serdes

Removing the dependency on serdes for the metastore requires a series of 
changes. I have created multiple commits which hopefully would be easier to 
review. Each major commit has a descriptive commit message to give a high level 
idea of what the change is doing. There are still some bits which need to be 
completed but it would be good to a review.

Overview of all the changes done:
1. Creates a new module called serde-api under storage-api like discussed. 
Although I think we can keep it separate as well.
2. Moved List, Map, Struct, Constant, Primitive, Union ObjectInspectors to 
serde-api
3. Moved PrimitiveTypeInfo, PrimitiveTypeEntry and TypeInfo to serde-api.
4. Moved TypeInfoParser, TypeInfoFactory to serde-api
5. Added a new class which reading avro storage schema by copying the code 
from AvroSerde and AvroSerdeUtils. The parsing is done such that String value 
is first converted into TypeInfos and then into FieldSchemas bypassing the need 
for ObjectInspectors. In theory we could get rid of TypeInfos as well but that 
path was getting too difficult with lot of duplicate code between Hive and 
metastore.
6. Introduces a default storage schema reader. I noticed that most of the 
serdes use the same logic to parse the metadata information. This code should 
be refactored to a common place instead of having many copies (one in 
standalone hms and another set in multiple serdes)
7. Moved HiveChar, HiveVarchar, HiveCharWritable, HiveVarcharWritable to 
storage-api. I noticed that HiveDecimal is already in storage-api. It probably 
makes sense to move the other primitive types (timestamp, interval etc)to 
storage-api as well but it requires storage-api to be upgraded to Java 8.
8. Adds a basic test for the schema reader. I plan to add more tests as 
this code is reviewed.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #287


commit bbfb7dc44904db74a840167c02b07f50a6010b69
Author: Vihang Karajgaonkar 
Date:   2017-11-09T00:52:39Z

HIVE-17580 : Remove dependency of get_fields_with_environment_context API 
to serde

commit d54879845eff10c19bc17bda9e09dda16f6fa295
Author: Vihang Karajgaonkar 
Date:   2017-11-29T00:54:23Z

Moved List, Map, Struct OI to storage-api

commit a12d6c7ba3de598c6b6f75da1bd4efcac43036b1
Author: Vihang Karajgaonkar 
Date:   2017-11-29T04:19:39Z

Moved ConstantObjectInspector PrimitiveObjectInspector and 
UnionObjectInspector

commit 13fb832fc2d51958e75d5e609f6781f87449aed8
Author: Vihang Karajgaonkar 
Date:   2017-12-28T01:25:59Z

Moved PrimitiveTypeInfo to serde-api

In order to move PrimitiveTypeInfo we need to move the PrimitiveTypeEntry 
as well.
PrimitiveTypeEntry depends on PrimitiveObjectInspectorUtils which cannot be 
pulled
into serde-api. Hence the static final maps are moved to PrimitiveEntry and 
we provide
static access methods to these maps along with the register method to add 
the key
value pairs in the maps

commit 9cbc789fd3f4ced7ce66a7313c451b75a154976f
Author: Vihang Karajgaonkar 
Date:   2017-12-28T20:51:39Z

Moved the other TypeInfos to serde-api

In order to move the other TypeInfo classes to serde-api we need to move 
the serdeConstants.java
as well. This is a thrift generated class. This commit copies the 
serde.thrift instead of moving.
The only reason I did not move it is in case of backwards compatibility 
reasons (in case someone is using the thrift file location to do something).
If it is okay to move serde.thrift from serde module to serde-api we can 
delete it from serde module in a separate
change.

The other concern is there are some TypeInfo classes which do some 
validation like VarCharTypeInfo, DecimalTypeInfo.
The validating methods use the actual type implementation like HiveChar, 
HiveDecimal etc to ensure that the params
are under the correct limits. This creates a problem since we cannot bring 
in the type implementations as well to
serde-api. Currently, I have marked these as TODO and commented them out.

commit 23aa899a90d17648e560d39602a3bd29bf53661e
Author: Vihang Karajgaonkar 
Date:   

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275877#comment-16275877
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


[~sershe], [~alangates], [~owen.omalley], [~akolb] and myself had a offline 
discussion over phone on this and we agreed to approach this problem with the 
following next steps:

1. Standalone-metastore should not have compile time dependency with SerDe and 
ObjectInspector.
In order to resolve the case for Avro, we will write a parser using Avro API to 
directly parse the file/url/property which returns the FieldSchema without 
using the SerDe. This parser will implement the interface StorageSchemaReader 
(see (3)). This means we may need to duplicate some of the AvroSerde's logic 
and return List for the given schema.url. This code cannot depend 
on OI, Deserializer and TypeInfo.

2. If the table belongs to one of the serdes listed in 
"hive.serdes.using.metastore.for.schema" metastore will return the FieldSchema 
from the DB (table.getSd().getCols()). 
This is similar to what we have currently.  In addition to the current 
implementation we should expand this list to include all the serdes defined in 
Hive source code.

3. If there is a table/partition belonging to a custom serde or to a serde 
which doesn't belong to the config listed in (2) standalone-metastore will use 
the interface StorageSchemaReader to read the schema. 
End-users are responsible for adding the jars in the metastore's classpath so 
that these custom serdes work with metastore using this interface.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270239#comment-16270239
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


After moving Deserializer, Serializer, AbstractSerde, ObjectInspector to a new 
module in storage-api I realized that I also needed to move the ListOI, MapOI, 
StructOI so that the existing code of MetastoreUtils.getFieldsFromDeserializer 
compiles when moved to standalone MS. For consistency I attempted to move 
ConstantOI, UnionOI and  PrimitiveObjectInspector as well to a new module 
within storage-api. Moving ConstantOI and UnionOI was trivial but moving 
PrimitiveObjectInspector brings along with it whole lot of other stuff. 
PrimitiveObjectInspector needs PrimitiveTypeInfo which needs PrimitiveTypeEntry 
to make the below method work. 
PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName gets the cached 
PrimitiveEntry objects for each primitive types and they bring in all the type 
implementations. Not to mention the TypeInfo needed for PrimitiveTypeInfo. 

{noformat}
public PrimitiveTypeEntry getPrimitiveTypeEntry() {
return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
  }
{noformat}

[~alangates] any thoughts or suggestions? I don't think we can make this API 
work without breaking compatibilities. Even if we want to create a compile time 
dependency on Deserializer while keeping the jar(s) in the classpath like 
[~sershe] suggested in HIVE-17714 we still need compile time dependency with 
ListOI, StructOI, MapOI with the current code to get fields from Deserializer. 

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179431#comment-16179431
 ] 

Alan Gates commented on HIVE-17580:
---

I'm +1 on using the SARGs for this, but I think we will have to continue to 
provide the current version of get_fields_with_environment_context for 
backwards compatibility.  So we should find something that works for it as 
well.  It makes sense to do that first and add a SARGs version of the call 
later.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)