Show You created HIVE-3335: ------------------------------ Summary: Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat Key: HIVE-3335 URL: https://issues.apache.org/jira/browse/HIVE-3335 Project: Hive Issue Type: Bug Components: Clients Affects Versions: 0.8.1 Environment: CentOS 5.8 x64 CDH3u4 hadoop-0.20-0.20.2+923.256-1 hadoop-0.20-{namenode,secondarynamenode,jobtracker,tasktracker,datanode}-0.20.2+923.256-1 hadoop-0.20-conf-pseudo-0.20.2+923.256-1(but same error was occurred on not pseudo env) apache hive-0.8.1(but same error was occurred on hive 0.9) Reporter: Show You
Procedure for reproduction: 1. Set up hadoop 2. Prepare data file and link.txt: data: $ hadoop fs -cat /path/to/data/2012-07-01/20120701.csv 1, 20120701 00:00:00 2, 20120701 00:00:01 3, 20120701 01:12:45 link.txt $ cat link.txt /path/to/data/2012-07-01//* 2. On hive, create table like below: CREATE TABLE user_logs(id INT, created_at STRING) row format delimited fields terminated by ',' lines terminated by '\n' stored as inputformat 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'; 3. Put link.txt to /user/hive/warehouse/user_logs $ sudo -u hdfs hadoop fs -put link.txt /user/hive/warehouse/user_logs 4. Open another session(A session), and watch socket, $ netstat -a | grep CLOSE_WAIT tcp 1 0 localhost:48121 localhost:50010 CLOSE_WAIT tcp 1 0 localhost:48124 localhost:50010 CLOSE_WAIT $ 5. Return to hive session, execute this, $ select * from user_logs; 6. Return to A session, watch socket again, $ netstat -a | grep CLOSE_WAIT tcp 1 0 localhost:48121 localhost:50010 CLOSE_WAIT tcp 1 0 localhost:48124 localhost:50010 CLOSE_WAIT tcp 1 0 localhost:48166 localhost:50010 CLOSE_WAIT If you makes any partitions, you'll watch unclosed socket whose count equals partitions by once. I think that this problem maybe is caused by this point: At https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java, line 66. BufferedReader was opened, but it doesn't closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira