[
https://issues.apache.org/jira/browse/HIVE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879441#action_12879441
]
Arvind Prabhakar commented on HIVE-1176:
----------------------------------------
bq. Can you elaborate on what you mean by 'some collections were being fetched
as semi-populated proxies with missing session context leading to NPEs'? Is
there something I can do to reproduce this?
@Paul: Here are the steps to reproduce this problem:
# Startout with a clean workspace checkout and apply the updated patch
HIVE-1176-2.patch.
# Manually revert the file
{{metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java}}
to its previous state
# run {{ant package}} from the root of the workspace
# run {{ant test}} from within metastore
You should see failures like the following:
{code}
[junit] testPartition() failed.
[junit] java.lang.NullPointerException
[junit] at
org.datanucleus.store.mapped.scostore.AbstractMapStore.validateKeyForWriting(AbstractMapStore.java:333)
[junit] at
org.datanucleus.store.mapped.scostore.JoinMapStore.put(JoinMapStore.java:252)
[junit] at org.datanucleus.sco.backed.Map.put(Map.java:640)
[junit] at
org.apache.hadoop.hive.metastore.api.Table.putToParameters(Table.java:359)
[junit] at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table(HiveMetaStore.java:1281)
[junit] at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:140)
[junit] at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testAlterTable(TestHiveMetaStore.java:728)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{code}
If you look at
{{src/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java}} you would
notice that the line causing this exception should ideally be a {{HashMap}} and
not an {{org.datanucleus.store.mapped.scostore.AbstractMapStore}} as indicated
by the stack trace. This happens because the datanucleus JDO framework replaces
collections with its own implementation in order to allow lazy-dereferencing
and optimize for database connections/queries/memory consumption etc.
Lazy loading of collections (and second class objects in general) can be
disabled at a global level or at entity level. Disabling this globally is
generally not recommended unless there is evidence backed by extensive testing
that supports that change. Disabling at an entity level is still OK provided
the entity object graph is fully dereferenced at all times. This could lead to
extensive memory consumption in the system in case the entity graph is huge.
My approach towards fixing the problem was to *not* change the default behavior
in the general case. Instead I felt that it was better to circumvent this
problem in the case of a remote metastore by creating a copy explicitly. If you
have other suggestions on how to address this, please let me know.
Also - more information on the lazy dereferencing mechanism used by datanucleus
framework can be found [here|http://www.datanucleus.org/plugins/core/sco.html].
> 'create if not exists' fails for a table name with 'select' in it
> -----------------------------------------------------------------
>
> Key: HIVE-1176
> URL: https://issues.apache.org/jira/browse/HIVE-1176
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore, Query Processor
> Reporter: Prasad Chakka
> Assignee: Arvind Prabhakar
> Fix For: 0.6.0
>
> Attachments: HIVE-1176-1.patch, HIVE-1176-2.patch,
> HIVE-1176.lib-files.tar.gz, HIVE-1176.patch
>
>
> hive> create table if not exists tmp_select(s string, c string, n int);
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
> exception: javax.jdo.JDOUserException JDOQL Single-String query should always
> start with SELECT)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesForDb(Hive.java:441)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:423)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:5538)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5192)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
> at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: MetaException(message:Got exception: javax.jdo.JDOUserException
> JDOQL Single-String query should always start with SELECT)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:612)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:450)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesForDb(Hive.java:439)
> ... 15 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
