[jira] [Created] (LENS-907) Subsequent calls to metastore API to fetch native tables throws error after the first call

[Deepak Barr (JIRA)]

Deepak Barr created LENS-907:
--------------------------------

             Summary: Subsequent calls to metastore API to fetch native tables 
throws error after the first call
                 Key: LENS-907
                 URL: https://issues.apache.org/jira/browse/LENS-907
             Project: Apache Lens
          Issue Type: Bug
            Reporter: Deepak Barr
            Assignee: Deepak Barr


{code:title=HiveSessionImpl.java}

  @Override
  public IMetaStoreClient getMetaStoreClient() throws HiveSQLException {
    if (metastoreClient == null) {
      try {
        metastoreClient = new HiveMetaStoreClient(getHiveConf());
      } catch (MetaException e) {
        throw new HiveSQLException(e);
      }
    }
    return metastoreClient;
  }
{code}

{code:title=CubeMetastoreServiceImpl.java}
private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, String 
dbName, boolean prependDbName)
    throws LensException {
    IMetaStoreClient msc = null;
    try {
      msc = getSession(sessionid).getMetaStoreClient();
      List<String> tables = msc.getAllTables(
        dbName);
      List<String> result = new ArrayList<String>();
      if (tables != null && !tables.isEmpty()) {
        List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
          msc.getTableObjectsByName(dbName, tables);
        Iterator<org.apache.hadoop.hive.metastore.api.Table> it = 
tblObjects.iterator();
        while (it.hasNext()) {
          org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
          if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == 
null) {
            if (prependDbName) {
              result.add(dbName + "." + tbl.getTableName());
            } else {
              result.add(tbl.getTableName());
            }
          }
        }
      }
      return result;
    } catch (Exception e) {
      throw new LensException("Error getting native tables from DB", e);
    } finally {
      if (null != msc) {
        msc.close();
      }
    }
  }

{code}

Scenario :

When we call the metastore resource API to fetch nativetables in a db, The 
getNativeTablesFromDB() method fetches the list of tables using 
(msc)HiveMetaStoreClient instance in HiveSessionImpl(). In finally, the msc 
instance is closed. The first call will succeed and return the list of tables.
Now, if we call the API again to fetch nativetables, the same instance of 
HiveMetaStoreClient which was closed in previous call. This results in the 
following error on msc.getAllTables() method call.

org.apache.thrift.transport.TTransportException: Cannot write to null 
outputStream
        at 
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
 ~[libthrift-0.9.0.jar:0.9.0]
        at 
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) 
~[libthrift-0.9.0.jar:0.9.0]
        at 
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
 ~[libthrift-0.9.0.jar:0.9.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) 
~[libthrift-0.9.0.jar:0.9.0]
        at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:538)
 ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:530)
 ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:970)
 ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[na:1.8.0_45]
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[na:1.8.0_45]
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
 ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at com.sun.proxy.$Proxy7.getDatabase(Unknown Source) [na:na]
        at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1156) 
[hive-exec-0.13.3-inm.jar:0.13.3-inm]
        at 
org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1145) 
[hive-exec-0.13.3-inm.jar:0.13.3-inm]
        at 
org.apache.lens.server.metastore.CubeMetastoreServiceImpl.getAllNativeTableNames(CubeMetastoreServiceImpl.java:1229)
 [classes/:na]

Solution :

There could be 2 solutions I can think of -

1. Do not close HiveMetaStoreClient object in getNativeTablesFromDB(). ( This 
was the case before LENS-796 )
2. Apply HIVE-8320 patch to inmobi hive codebase. We applied the patch for our 
system and It solves the problem.


Any thoughts ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)