[
https://issues.apache.org/jira/browse/HIVE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20627:
------------------------------------
Description:
When multiple async queries are executed from same session, it leads to
multiple async query execution DAGs share the same Hive object which is set by
caller for all threads. In case of loading dynamic partitions, it creates
MoveTask which re-creates the Hive object and closes the shared Hive object
which causes metastore connection issues for other async execution thread who
still access it. This is also seen if ReplDumpTask and ReplLoadTask are part of
the DAG.
*Call Stack:*
{code:java}
2018-09-16T04:38:04,280 ERROR [load-dynamic-partitions-7]: metadata.Hive
(Hive.java:call(2436)) - Exception when loading partition with parameters
partPath=hdfs://mycluster/warehouse/tablespace/managed/hive/tbl_3bcvvdubni/.hive-staging_hive_2018-09-16_04-35-50_708_7776079613819042057-1147/-ext-10000/age=55,
table=tbl_3bcvvdubni, partSpec={age=55}, loadFileType=KEEP_EXISTING,
listBucketingLevel=0, isAcid=true, hasFollowingStatsTask=true
org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the
metastore
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:714)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.ql.io.AcidUtils.getTableValidWriteIdListWithTxnList(AcidUtils.java:1791)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1756)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1714)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1976)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2415)
[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2406)
[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.apache.thrift.protocol.TProtocolException: Required field
'validTxnList' is unset!
Struct:GetValidWriteIdsRequest(fullTableNames:[default.tbl_3bcvvdubni],
validTxnList:null)
at
org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest.validate(GetValidWriteIdsRequest.java:396)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.validate(ThriftHiveMetastore.java)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.write(ThriftHiveMetastore.java)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_valid_write_ids(ThriftHiveMetastore.java:5443)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_valid_write_ids(ThriftHiveMetastore.java:5435)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2589)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at com.sun.proxy.$Proxy57.getValidWriteIds(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2934)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
at com.sun.proxy.$Proxy57.getValidWriteIds(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:712)
~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
... 10 more{code}
*Root cause:*
For Async query execution from SQLOperation.runInternal, we set the Thread
local Hive object for all the child threads as parentHive
(parentSession.getSessionHive())
{code:java}
@Override
public void run() {
PrivilegedExceptionAction<Object> doAsAction = new
PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws HiveSQLException {
Hive.set(parentHive); // Setting parentHive for all async operations.
// TODO: can this result in cross-thread reuse of session state?
SessionState.setCurrentSessionState(parentSessionState);
PerfLogger.setPerfLogger(parentPerfLogger);
LogUtils.registerLoggingContext(queryState.getConf());
try {
if (asyncPrepare) {
prepare(queryState);
}
runQuery();
} catch (HiveSQLException e) {
// TODO: why do we invent our own error path op top of the one from Future.get?
setOperationException(e);
LOG.error("Error running hive query: ", e);
} finally {
LogUtils.unregisterLoggingContext();
}
return null;
}
};
{code}
Now, when async execution in progress and if one of the thread re-creates the
Hive object, it closes the parentHive object first which impacts other threads
using it and hence conf object it refers too gets cleaned up and hence we get
null for VALID_TXNS_KEY value.
{code:java}
private static Hive create(HiveConf c, boolean needsRefresh, Hive db, boolean
doRegisterAllFns)
throws HiveException {
if (db != null) {
LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + needsRefresh +
", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
db.close();
}
closeCurrent();
if (c == null) {
c = createHiveConf();
}
c.set("fs.scheme.class", "dfs");
Hive newdb = new Hive(c, doRegisterAllFns);
hiveDB.set(newdb);
return newdb;
}
{code}
*Fix:*
We shouldn't clean the old Hive object if it is shared by multiple threads.
Shall use a flag to know this.
*Memory leak issue:*
Memory leak is found if one of the threads from Hive.loadDynamicPartitions
throw exception. rawStoreMap is used to store rawStore objects which has to be
cleaned. In this case, it is populated only in success flow but if there are
exceptions, it is not and hence there is a leak.
{code:java}
futures.add(pool.submit(new Callable<Void>() {
@Override
public Void call() throws Exception {
try {
// move file would require session details (needCopy() invokes
SessionState.get)
SessionState.setCurrentSessionState(parentSession);
LOG.info("New loading path = " + partPath + " with partSpec " + fullPartSpec);
// load the partition
Partition newPartition = loadPartition(partPath, tbl, fullPartSpec,
loadFileType,
true, false, numLB > 0, false, isAcid, hasFollowingStatsTask, writeId, stmtId,
isInsertOverwrite);
partitionsMap.put(fullPartSpec, newPartition);
if (inPlaceEligible) {
synchronized (ps) {
InPlaceUpdate.rePositionCursor(ps);
partitionsLoaded.incrementAndGet();
InPlaceUpdate.reprintLine(ps, "Loaded : " + partitionsLoaded.get() + "/"
+ partsToLoad + " partitions.");
}
}
// Add embedded rawstore, so we can cleanup later to avoid memory leak
if (getMSC().isLocalMetaStore()) {
if (!rawStoreMap.containsKey(Thread.currentThread().getId())) {
rawStoreMap.put(Thread.currentThread().getId(),
HiveMetaStore.HMSHandler.getRawStore());
}
}
return null;
} catch (Exception t) {
}
{code}
was:
When multiple async queries are executed from same session, it leads to
multiple async query execution DAGs share the same Hive object which is set by
caller for all threads. In case of loading dynamic partitions, it creates
MoveTask which re-creates the Hive object and closes the shared Hive object
which causes metastore connection issues for other async execution thread who
still access it. This is also seen if ReplDumpTask and ReplLoadTask are part of
the DAG.
*Root cause:*
For Async query execution from SQLOperation.runInternal, we set the Thread
local Hive object for all the child threads as parentHive
(parentSession.getSessionHive())
{code}
@Override
public void run() {
PrivilegedExceptionAction<Object> doAsAction = new
PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws HiveSQLException {
Hive.set(parentHive); // Setting parentHive for all async operations.
// TODO: can this result in cross-thread reuse of session state?
SessionState.setCurrentSessionState(parentSessionState);
PerfLogger.setPerfLogger(parentPerfLogger);
LogUtils.registerLoggingContext(queryState.getConf());
try {
if (asyncPrepare) {
prepare(queryState);
}
runQuery();
} catch (HiveSQLException e) {
// TODO: why do we invent our own error path op top of the one from Future.get?
setOperationException(e);
LOG.error("Error running hive query: ", e);
} finally {
LogUtils.unregisterLoggingContext();
}
return null;
}
};
{code}
Now, when async execution in progress and if one of the thread re-creates the
Hive object, it closes the parentHive object first which impacts other threads
using it and hence conf object it refers too gets cleaned up and hence we get
null for VALID_TXNS_KEY value.
{code}
private static Hive create(HiveConf c, boolean needsRefresh, Hive db, boolean
doRegisterAllFns)
throws HiveException {
if (db != null) {
LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + needsRefresh +
", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
db.close();
}
closeCurrent();
if (c == null) {
c = createHiveConf();
}
c.set("fs.scheme.class", "dfs");
Hive newdb = new Hive(c, doRegisterAllFns);
hiveDB.set(newdb);
return newdb;
}
{code}
*Fix:*
We shouldn't clean the old Hive object if it is shared by multiple threads.
Shall use a flag to know this.
*Memory leak issue:*
Memory leak is found if one of the threads from Hive.loadDynamicPartitions
throw exception. rawStoreMap is used to store rawStore objects which has to be
cleaned. In this case, it is populated only in success flow but if there are
exceptions, it is not and hence there is a leak.
{code}
futures.add(pool.submit(new Callable<Void>() {
@Override
public Void call() throws Exception {
try {
// move file would require session details (needCopy() invokes
SessionState.get)
SessionState.setCurrentSessionState(parentSession);
LOG.info("New loading path = " + partPath + " with partSpec " + fullPartSpec);
// load the partition
Partition newPartition = loadPartition(partPath, tbl, fullPartSpec,
loadFileType,
true, false, numLB > 0, false, isAcid, hasFollowingStatsTask, writeId, stmtId,
isInsertOverwrite);
partitionsMap.put(fullPartSpec, newPartition);
if (inPlaceEligible) {
synchronized (ps) {
InPlaceUpdate.rePositionCursor(ps);
partitionsLoaded.incrementAndGet();
InPlaceUpdate.reprintLine(ps, "Loaded : " + partitionsLoaded.get() + "/"
+ partsToLoad + " partitions.");
}
}
// Add embedded rawstore, so we can cleanup later to avoid memory leak
if (getMSC().isLocalMetaStore()) {
if (!rawStoreMap.containsKey(Thread.currentThread().getId())) {
rawStoreMap.put(Thread.currentThread().getId(),
HiveMetaStore.HMSHandler.getRawStore());
}
}
return null;
} catch (Exception t) {
}
{code}
> Concurrent async queries intermittently fails with LockException and cause
> memory leak.
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-20627
> URL: https://issues.apache.org/jira/browse/HIVE-20627
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.0.0, 3.2.0
> Reporter: Sankar Hariappan
> Assignee: Sankar Hariappan
> Priority: Major
>
> When multiple async queries are executed from same session, it leads to
> multiple async query execution DAGs share the same Hive object which is set
> by caller for all threads. In case of loading dynamic partitions, it creates
> MoveTask which re-creates the Hive object and closes the shared Hive object
> which causes metastore connection issues for other async execution thread who
> still access it. This is also seen if ReplDumpTask and ReplLoadTask are part
> of the DAG.
> *Call Stack:*
> {code:java}
> 2018-09-16T04:38:04,280 ERROR [load-dynamic-partitions-7]: metadata.Hive
> (Hive.java:call(2436)) - Exception when loading partition with parameters
> partPath=hdfs://mycluster/warehouse/tablespace/managed/hive/tbl_3bcvvdubni/.hive-staging_hive_2018-09-16_04-35-50_708_7776079613819042057-1147/-ext-10000/age=55,
> table=tbl_3bcvvdubni, partSpec={age=55}, loadFileType=KEEP_EXISTING,
> listBucketingLevel=0, isAcid=true, hasFollowingStatsTask=true
> org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the
> metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableValidWriteIdListWithTxnList(AcidUtils.java:1791)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1756)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1976)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2415)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2406)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_171]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> Caused by: org.apache.thrift.protocol.TProtocolException: Required field
> 'validTxnList' is unset!
> Struct:GetValidWriteIdsRequest(fullTableNames:[default.tbl_3bcvvdubni],
> validTxnList:null)
> at
> org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest.validate(GetValidWriteIdsRequest.java:396)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.validate(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.write(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_valid_write_ids(ThriftHiveMetastore.java:5443)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_valid_write_ids(ThriftHiveMetastore.java:5435)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2589)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_171]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at com.sun.proxy.$Proxy57.getValidWriteIds(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_171]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2934)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at com.sun.proxy.$Proxy57.getValidWriteIds(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:712)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> ... 10 more{code}
> *Root cause:*
> For Async query execution from SQLOperation.runInternal, we set the Thread
> local Hive object for all the child threads as parentHive
> (parentSession.getSessionHive())
> {code:java}
> @Override
> public void run() {
> PrivilegedExceptionAction<Object> doAsAction = new
> PrivilegedExceptionAction<Object>() {
> @Override
> public Object run() throws HiveSQLException {
> Hive.set(parentHive); // Setting parentHive for all async operations.
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(parentPerfLogger);
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // TODO: why do we invent our own error path op top of the one from
> Future.get?
> setOperationException(e);
> LOG.error("Error running hive query: ", e);
> } finally {
> LogUtils.unregisterLoggingContext();
> }
> return null;
> }
> };
> {code}
> Now, when async execution in progress and if one of the thread re-creates the
> Hive object, it closes the parentHive object first which impacts other
> threads using it and hence conf object it refers too gets cleaned up and
> hence we get null for VALID_TXNS_KEY value.
> {code:java}
> private static Hive create(HiveConf c, boolean needsRefresh, Hive db, boolean
> doRegisterAllFns)
> throws HiveException {
> if (db != null) {
> LOG.debug("Creating new db. db = " + db + ", needsRefresh = " + needsRefresh
> +
> ", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
> db.close();
> }
> closeCurrent();
> if (c == null) {
> c = createHiveConf();
> }
> c.set("fs.scheme.class", "dfs");
> Hive newdb = new Hive(c, doRegisterAllFns);
> hiveDB.set(newdb);
> return newdb;
> }
> {code}
> *Fix:*
> We shouldn't clean the old Hive object if it is shared by multiple threads.
> Shall use a flag to know this.
> *Memory leak issue:*
> Memory leak is found if one of the threads from Hive.loadDynamicPartitions
> throw exception. rawStoreMap is used to store rawStore objects which has to
> be cleaned. In this case, it is populated only in success flow but if there
> are exceptions, it is not and hence there is a leak.
> {code:java}
> futures.add(pool.submit(new Callable<Void>() {
> @Override
> public Void call() throws Exception {
> try {
> // move file would require session details (needCopy() invokes
> SessionState.get)
> SessionState.setCurrentSessionState(parentSession);
> LOG.info("New loading path = " + partPath + " with partSpec " +
> fullPartSpec);
> // load the partition
> Partition newPartition = loadPartition(partPath, tbl, fullPartSpec,
> loadFileType,
> true, false, numLB > 0, false, isAcid, hasFollowingStatsTask, writeId,
> stmtId,
> isInsertOverwrite);
> partitionsMap.put(fullPartSpec, newPartition);
> if (inPlaceEligible) {
> synchronized (ps) {
> InPlaceUpdate.rePositionCursor(ps);
> partitionsLoaded.incrementAndGet();
> InPlaceUpdate.reprintLine(ps, "Loaded : " + partitionsLoaded.get() + "/"
> + partsToLoad + " partitions.");
> }
> }
> // Add embedded rawstore, so we can cleanup later to avoid memory leak
> if (getMSC().isLocalMetaStore()) {
> if (!rawStoreMap.containsKey(Thread.currentThread().getId())) {
> rawStoreMap.put(Thread.currentThread().getId(),
> HiveMetaStore.HMSHandler.getRawStore());
> }
> }
> return null;
> } catch (Exception t) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
