[
https://issues.apache.org/jira/browse/LENS-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070965#comment-15070965
]
Hudson commented on LENS-907:
-----------------------------
SUCCESS: Integrated in Lens-Commit #1161 (See
[https://builds.apache.org/job/Lens-Commit/1161/])
LENS-907 : Subsequent calls to metastore API to fetch native tables
(deepakbarr: rev 4d7c8e4db3e9aa81da042921a31dfd670982ce38)
*
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java
> Subsequent calls to metastore API to fetch native tables throws error after
> the first call
> ------------------------------------------------------------------------------------------
>
> Key: LENS-907
> URL: https://issues.apache.org/jira/browse/LENS-907
> Project: Apache Lens
> Issue Type: Bug
> Reporter: Deepak Barr
> Assignee: Deepak Barr
> Attachments: LENS-907.patch
>
>
> {code:title=HiveSessionImpl.java}
> @Override
> public IMetaStoreClient getMetaStoreClient() throws HiveSQLException {
> if (metastoreClient == null) {
> try {
> metastoreClient = new HiveMetaStoreClient(getHiveConf());
> } catch (MetaException e) {
> throw new HiveSQLException(e);
> }
> }
> return metastoreClient;
> }
> {code}
> {code:title=CubeMetastoreServiceImpl.java}
> private List<String> getNativeTablesFromDB(LensSessionHandle sessionid,
> String dbName, boolean prependDbName)
> throws LensException {
> IMetaStoreClient msc = null;
> try {
> msc = getSession(sessionid).getMetaStoreClient();
> List<String> tables = msc.getAllTables(
> dbName);
> List<String> result = new ArrayList<String>();
> if (tables != null && !tables.isEmpty()) {
> List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
> msc.getTableObjectsByName(dbName, tables);
> Iterator<org.apache.hadoop.hive.metastore.api.Table> it =
> tblObjects.iterator();
> while (it.hasNext()) {
> org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
> if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) ==
> null) {
> if (prependDbName) {
> result.add(dbName + "." + tbl.getTableName());
> } else {
> result.add(tbl.getTableName());
> }
> }
> }
> }
> return result;
> } catch (Exception e) {
> throw new LensException("Error getting native tables from DB", e);
> } finally {
> if (null != msc) {
> msc.close();
> }
> }
> }
> {code}
> Scenario :
> When we call the metastore resource API to fetch nativetables in a db, The
> getNativeTablesFromDB() method fetches the list of tables using
> (msc)HiveMetaStoreClient instance in HiveSessionImpl(). In finally, the msc
> instance is closed. The first call will succeed and return the list of tables.
> Now, if we call the API again to fetch nativetables, the same instance of
> HiveMetaStoreClient which was closed in previous call. This results in the
> following error on msc.getAllTables() method call.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
> ~[libthrift-0.9.0.jar:0.9.0]
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
> ~[libthrift-0.9.0.jar:0.9.0]
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
> ~[libthrift-0.9.0.jar:0.9.0]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> ~[libthrift-0.9.0.jar:0.9.0]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:538)
> ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:530)
> ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:970)
> ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_45]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
> ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at com.sun.proxy.$Proxy7.getDatabase(Unknown Source) [na:na]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1156)
> [hive-exec-0.13.3-inm.jar:0.13.3-inm]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1145)
> [hive-exec-0.13.3-inm.jar:0.13.3-inm]
> at
> org.apache.lens.server.metastore.CubeMetastoreServiceImpl.getAllNativeTableNames(CubeMetastoreServiceImpl.java:1229)
> [classes/:na]
> Solution :
> There could be 2 solutions I can think of -
> 1. Do not close HiveMetaStoreClient object in getNativeTablesFromDB(). ( This
> was the case before LENS-796 )
> 2. Apply HIVE-8320 patch to inmobi hive codebase. We applied the patch for
> our system and It solves the problem.
> Any thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
