Iuliia Emelianova created HBASE-25581:
-----------------------------------------
Summary: BUG - HBase native client memory leak - threads keep
running after connection is closed
Key: HBASE-25581
URL: https://issues.apache.org/jira/browse/HBASE-25581
Project: HBase
Issue Type: Sub-task
Reporter: Iuliia Emelianova
Hello everyone,
First of all, thank you for creating such an amazing product
I'm trying to implement an open-source PHP-extension based on HBase native
client. It seems more promising performance-wise than Thrifts we're currently
using
In terms of PHP-fpm, we need to be able to open and close connection multiple
times in one process, and here's when the library leaks memory
Calling 'Close' methods on Table and Clients is not enough for closing all of
the threads.
A simple program
{code:java}
int main() {
zoo_set_debug_level( (ZooLogLevel)4);
google::SetCommandLineOption("GLOG_minloglevel", "0");
std::shared_ptr<hbase::Configuration> conf =
std::make_shared<hbase::Configuration>();
conf->Set(hbase::ZKUtil::kHBaseZookeeperQuorum_, "quorum");
conf->Set(hbase::ZKUtil::kHBaseZookeeperClientPort_, "2181");
conf->Set(hbase::ZKUtil::kHBaseZnodeParent_, "/hbase-unsecure");
createClient(conf);
int i = 0;
while (true) {
i++;
std::cout << "Slept " << i << std::endl;
sleep(5);
}
return 0;
}
void createClient(const std::shared_ptr<hbase::Configuration>& conf)
{
std::cout << "opening " << std::endl << std::endl << std::endl;
auto tn =
std::make_shared<hbase::pb::TableName>(folly::to<hbase::pb::TableName>("table"));
auto client = std::make_unique<hbase::Client>(*conf);
auto table = client->Table(*tn);
std::cout << "closing " << std::endl << std::endl << std::endl;
table->Close();
client->Close();
}
{code}
Logs :
{code:java}
[julia@fdf346c7aa2b myhbase]$ ./myhbase
opening
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0216 08:54:44.391023 5259 location-cache.cc:77] Connecting to ZooKeeper.
Quorum:quorum
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@726: Client
environment:zookeeper.version=zookeeper C client 3.4.8
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@730: Client
environment:host.name=fdf346c7aa2b
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@737: Client
environment:os.name=Linux
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@738: Client
environment:os.arch=5.8.1-1.el7.elrepo.x86_64
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@739: Client
environment:os.version=#1 SMP Tue Aug 11 12:01:11 EDT 2020
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@747: Client
environment:user.name=(null)
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@755: Client
environment:user.home=/home/julia
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@767: Client
environment:user.dir=/usr/src/hbase/myhbase
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@zookeeper_init@800:
Initiating client connection, host=quorum sessionTimeout=90000 watcher=(nil)
sessionId=0 sessionPasswd=<null> context=(nil) flags=0
2021-02-16 08:54:44,400:5259(0x7fa88b38d3c0):ZOO_DEBUG@start_threads@221:
starting threads...
2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_DEBUG@do_io@367: started IO
thread
2021-02-16 08:54:44,400:5259(0x7fa7a6ffd700):ZOO_DEBUG@do_completion@459:
started completion thread
closing
I0216 08:54:44.400823 5259 client.cc:56] called destructor on Client
2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1728:
initiated connection to server [192.168.197.12:2181]
2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1775:
session establishment complete on server [192.168.197.12:2181],
sessionId=0x27640b8a9bd0932, negotiated timeout=60000
2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_DEBUG@check_events@1781:
Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE
2021-02-16
08:54:44,403:5259(0x7fa7a6ffd700):ZOO_DEBUG@process_completions@2132: Calling a
watcher for node [], type = -1 event=ZOO_SESSION_EVENT
I0216 08:54:44.403533 5259 table.cc:50] called destructor on Table
Slept 1
Slept 2
Slept 3
Slept 4
Slept 5
2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570:
Exceeded deadline by 19ms
2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218:
Got ping response in 0 ms
Slept 6
Slept 7
Slept 8
Slept 9
2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570:
Exceeded deadline by 14ms
2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218:
Got ping response in 0 ms
Slept 10
Slept 11
Slept 12
{code}
As you can see, after calling destructors on Table and Client, a connection to
Zookeeper is still open, no other destructors are called.
If I'm calling createClient() function multiple times, I can see consumed
memory is growing and being released only after the program exits
Best regards,
Julia
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
