[jira] [Created] (HIVE-13612) Suppress error message of which command when hbase binary is absent
Kousuke Saruta created HIVE-13612: - Summary: Suppress error message of which command when hbase binary is absent Key: HIVE-13612 URL: https://issues.apache.org/jira/browse/HIVE-13612 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 2.1.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta Priority: Trivial hive command try finding hbase binary by which command and if the binary is absent, which command print error message. On the other hand, finding hadoop binary by which command redirects error message to /dev/null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13472) Replace primitive wrapper's valueOf method with parse* method to avoid unnecessary boxing/unboxing
Kousuke Saruta created HIVE-13472: - Summary: Replace primitive wrapper's valueOf method with parse* method to avoid unnecessary boxing/unboxing Key: HIVE-13472 URL: https://issues.apache.org/jira/browse/HIVE-13472 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 2.1.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta There are lots of primitive wrapper's valueOf method which should be replaced with parseXX method. For example, Integer.valueOf(String) returns Integer type but Integer.parseInt(String) returns primitive int type so we can avoid unnecessary boxing/unboxing by replacing some of them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13434) BaseSemanticAnalyzer.unescapeSQLString doesn't unescape \0000 style character literals.
Kousuke Saruta created HIVE-13434: - Summary: BaseSemanticAnalyzer.unescapeSQLString doesn't unescape \ style character literals. Key: HIVE-13434 URL: https://issues.apache.org/jira/browse/HIVE-13434 Project: Hive Issue Type: Bug Components: Parser Affects Versions: 2.1.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta BaseSemanticAnalyzer.unescapeSQLString method may have a fault. When "\u0061" style character literals are passed to the method, it's not unescaped successfully. In Spark SQL project, we referenced the unescaping logic and noticed this issue (SPARK-14426) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5833) Remove versions from child module dependencies
[
https://issues.apache.org/jira/browse/HIVE-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5833:
-
Status: Patch Available (was: Open)
Remove versions from child module dependencies
--
Key: HIVE-5833
URL: https://issues.apache.org/jira/browse/HIVE-5833
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Attachments: HIVE-5833.2.patch, HIVE-5833.patch
HIVE-5741 moved all dependencies to the plugin management section of the
parent pom therefore we can remove
{noformat}version${dep.version}/version{noformat} from all dependencies
in child modules.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5833) Remove versions from child module dependencies
[
https://issues.apache.org/jira/browse/HIVE-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5833:
-
Attachment: HIVE-5833.2.patch
[~brocknoland] Thank you for your comment!
I've just modified that.
Remove versions from child module dependencies
--
Key: HIVE-5833
URL: https://issues.apache.org/jira/browse/HIVE-5833
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Attachments: HIVE-5833.2.patch, HIVE-5833.patch
HIVE-5741 moved all dependencies to the plugin management section of the
parent pom therefore we can remove
{noformat}version${dep.version}/version{noformat} from all dependencies
in child modules.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5833) Remove versions from child module dependencies
[
https://issues.apache.org/jira/browse/HIVE-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5833:
-
Attachment: HIVE-5833.patch
I've tried to make a patch for this issue.
Remove versions from child module dependencies
--
Key: HIVE-5833
URL: https://issues.apache.org/jira/browse/HIVE-5833
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Attachments: HIVE-5833.patch
HIVE-5741 moved all dependencies to the plugin management section of the
parent pom therefore we can remove version${dep.version}/version from all
dependencies in child modules.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5268) HiveServer2 accumulates orphaned OperationHandle objects when a client fails while executing query
[ https://issues.apache.org/jira/browse/HIVE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814094#comment-13814094 ] Kousuke Saruta commented on HIVE-5268: -- I think this issue is related to HIVE-5296. HiveServer2 accumulates orphaned OperationHandle objects when a client fails while executing query -- Key: HIVE-5268 URL: https://issues.apache.org/jira/browse/HIVE-5268 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Vaibhav Gumashta Assignee: Thiruvel Thirumoolan Fix For: 0.13.0 Attachments: HIVE-5268_prototype.patch When queries are executed against the HiveServer2 an OperationHandle object is stored in the OperationManager.handleToOperation HashMap. Currently its the duty of the JDBC client to explicitly close to cleanup the entry in the map. But if the client fails to close the statement then the OperationHandle object is never cleaned up and gets accumulated in the server. This can potentially cause OOM on the server over time. This also can be used as a loophole by a malicious client to bring down the Hive server. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Fix Version/s: 0.13.0 JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Labels: patch Fix For: 0.13.0 Attachments: HIVE-2137.patch, HIVE-2137.patch, HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5385) StringUtils is not in commons codec 1.3
[ https://issues.apache.org/jira/browse/HIVE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786444#comment-13786444 ] Kousuke Saruta commented on HIVE-5385: -- [~yhuai] I think we should still keep 0.20.2 because there a little bit of problem of compatibility between 0.20.2 and 0.20.205+ ( e.g. UnixUserGroupInformation is no longer used in 0.20.205). OK, I'll try to remove dependency of 1.3. StringUtils is not in commons codec 1.3 --- Key: HIVE-5385 URL: https://issues.apache.org/jira/browse/HIVE-5385 Project: Hive Issue Type: Bug Reporter: Yin Huai Assignee: Kousuke Saruta Priority: Trivial Attachments: HIVE-5385.1.patch In ThriftHttpServlet introduced by HIVE-4763, StringUtils is imported which was introduced by commons codec 1.4. But, our 0.20 shims depends on commons codec 1.3. Our eclipse classpath template is also using libs of 0.20 shims. So, we will get two errors in eclipse. Compiling hive will not have a problem because we are loading codec 1.4 for project service (1.4 is also used when -Dhadoop.version=0.20.2 -Dhadoop.mr.rev=20). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5385) StringUtils is not in commons codec 1.3
[ https://issues.apache.org/jira/browse/HIVE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-5385: - Attachment: HIVE-5385.2.patch I've attached a 2nd patch for this issue. StringUtils is not in commons codec 1.3 --- Key: HIVE-5385 URL: https://issues.apache.org/jira/browse/HIVE-5385 Project: Hive Issue Type: Bug Reporter: Yin Huai Assignee: Kousuke Saruta Priority: Trivial Attachments: HIVE-5385.1.patch, HIVE-5385.2.patch In ThriftHttpServlet introduced by HIVE-4763, StringUtils is imported which was introduced by commons codec 1.4. But, our 0.20 shims depends on commons codec 1.3. Our eclipse classpath template is also using libs of 0.20 shims. So, we will get two errors in eclipse. Compiling hive will not have a problem because we are loading codec 1.4 for project service (1.4 is also used when -Dhadoop.version=0.20.2 -Dhadoop.mr.rev=20). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5385) StringUtils is not in commons codec 1.3
[ https://issues.apache.org/jira/browse/HIVE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785539#comment-13785539 ] Kousuke Saruta commented on HIVE-5385: -- [~yhuai] Sorry, I had a mistake. I found why commons-codec 1.3 is downloaded is because hadoop-core(0.20.2) depends on commons-codec 1.3. So, if we use commons-codec 1.4, we should use newer hadoop-core ( maybe 0.20.205 or 1.x) for hadoop-core. StringUtils is not in commons codec 1.3 --- Key: HIVE-5385 URL: https://issues.apache.org/jira/browse/HIVE-5385 Project: Hive Issue Type: Bug Reporter: Yin Huai Priority: Trivial Attachments: HIVE-5385.1.patch In ThriftHttpServlet introduced by HIVE-4763, StringUtils is imported which was introduced by commons codec 1.4. But, our 0.20 shims depends on commons codec 1.3. Our eclipse classpath template is also using libs of 0.20 shims. So, we will get two errors in eclipse. Compiling hive will not have a problem because we are loading codec 1.4 for project service (1.4 is also used when -Dhadoop.version=0.20.2 -Dhadoop.mr.rev=20). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5385) StringUtils is not in commons codec 1.3
[ https://issues.apache.org/jira/browse/HIVE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-5385: - Attachment: HIVE-5385.1.patch Hi, I've created a patch for this issue and confirmed to build successfully. StringUtils is not in commons codec 1.3 --- Key: HIVE-5385 URL: https://issues.apache.org/jira/browse/HIVE-5385 Project: Hive Issue Type: Bug Reporter: Yin Huai Priority: Trivial Attachments: HIVE-5385.1.patch In ThriftHttpServlet introduced by HIVE-4763, StringUtils is imported which was introduced by commons codec 1.4. But, our 0.20 shims depends on commons codec 1.3. Our eclipse classpath template is also using libs of 0.20 shims. So, we will get two errors in eclipse. Compiling hive will not have a problem because we are loading codec 1.4 for project service (1.4 is also used when -Dhadoop.version=0.20.2 -Dhadoop.mr.rev=20). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782075#comment-13782075
]
Kousuke Saruta commented on HIVE-5296:
--
Hi [~doug.mcilwraith]
My patch is for one of two causes of memory leak already known.
So, you should also watch HIVE-4501.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.2.patch, HIVE-5296.patch,
HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
[jira] [Commented] (HIVE-4501) HS2 memory leak - FileSystem objects in FileSystem.CACHE
[ https://issues.apache.org/jira/browse/HIVE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780156#comment-13780156 ] Kousuke Saruta commented on HIVE-4501: -- Hi [~apivovarov] I revased your patch and found that FileSystem.CACHE doesn't leak. HS2 memory leak - FileSystem objects in FileSystem.CACHE Key: HIVE-4501 URL: https://issues.apache.org/jira/browse/HIVE-4501 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Critical Attachments: HIVE-4501.1.patch, HIVE-4501.1.patch org.apache.hadoop.fs.FileSystem objects are getting accumulated in FileSystem.CACHE, with HS2 in unsecure mode. As a workaround, it is possible to set fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to true. Users should not have to bother with this extra configuration. As a workaround disable impersonation by setting hive.server2.enable.doAs to false. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) Cannot attach debugger to Hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Assignee: Kousuke Saruta
Cannot attach debugger to Hiveserver2
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
Kousuke Saruta created HIVE-5374:
Summary: hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double quote
keyword (colmn name, table name etc ), those name is treated case sensitively.
But in the script, there is a non double quoted table name and column anme
although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5374:
-
Attachment: HIVE-5374.patch.1
I've attached a patch for this issue.
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5374.patch.1
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those name is treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5374:
-
Status: Patch Available (was: Open)
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5374.patch.1
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those name is treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5374:
-
Description:
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double quote
keyword (colmn name, table name etc ), those names are treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
was:
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double quote
keyword (colmn name, table name etc ), those name is treated case sensitively.
But in the script, there is a non double quoted table name and column anme
although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5374.patch.1
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those names are treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779459#comment-13779459
]
Kousuke Saruta commented on HIVE-5296:
--
[~vgumashta] Thank you for your comment. I will submit next patch soon.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.patch, HIVE-5296.patch,
HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.patch.2
I've attached a new patch.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.patch, HIVE-5296.patch,
HIVE-5296.patch, HIVE-5296.patch.2
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient
[jira] [Updated] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5374:
-
Attachment: HIVE-5374.patch.2
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.12.0, 0.13.0
Attachments: HIVE-5374.patch.1, HIVE-5374.patch.2
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those names are treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779507#comment-13779507
]
Kousuke Saruta commented on HIVE-5374:
--
[~prasadm] sorry, I've already created a rebased patch before seeing your
latest post. So, could you review that?
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.12.0, 0.13.0
Attachments: HIVE-5374.patch.1, HIVE-5374.patch.2
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those names are treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.2.patch
Sorry, I submitted a wrong named patch. I've submitted a renamed patch.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.2.patch, HIVE-5296.patch,
HIVE-5296.patch, HIVE-5296.patch, HIVE-5296.patch.2
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: (was: HIVE-5296.patch.2)
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.2.patch, HIVE-5296.patch,
HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used
[jira] [Commented] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779551#comment-13779551
]
Kousuke Saruta commented on HIVE-5315:
--
Could anyone review the patch or give me comments?
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5374) hive-schema-0.13.0.postgres.sql doesn't work
[
https://issues.apache.org/jira/browse/HIVE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5374:
-
Attachment: HIVE-5374.1.patch
Sorry, I submitted a wrong named patch. I've submitted a new patch.
hive-schema-0.13.0.postgres.sql doesn't work
Key: HIVE-5374
URL: https://issues.apache.org/jira/browse/HIVE-5374
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.12.0, 0.13.0
Attachments: HIVE-5374.1.patch, HIVE-5374.patch.1, HIVE-5374.patch.2
hive-schema-0.13.0.postgres.sql doesn't work. In PostgreSQL, if we double
quote keyword (colmn name, table name etc ), those names are treated case
sensitively. But in the script, there is a non double quoted table name and
column anme although those are double quoted at the definition.
{code}
CREATE TABLE VERSION (
VER_ID bigint,
SCHEMA_VERSION character varying(127) NOT NULL,
COMMENT character varying(255) NOT NULL,
PRIMARY KEY (VER_ID)
);
{code}
{code}
INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1,
'0.13.0', 'Hive release version 0.13.0');
{code}
Also, the definition above defines column COMMENT but I think it should be
named VERSION_COMMENT.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) Cannot attach debugger to Hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Summary: Cannot attach debugger to Hiveserver2 (was: bin/hive should
retrieve HADOOP_VERSION by better way.)
Cannot attach debugger to Hiveserver2
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.patch
I've attached a modified patch.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch, HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Affects Version/s: 0.13.0
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch, HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar process is seen to increase
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777901#comment-13777901
]
Kousuke Saruta commented on HIVE-5296:
--
The error above may be caused by HIVE-5360. So, I've re-submitted a patch.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.patch, HIVE-5296.patch,
HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.1.patch
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.1.patch, HIVE-5296.patch, HIVE-5296.patch,
HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar
[jira] [Commented] (HIVE-4501) HS2 memory leak - FileSystem objects in FileSystem.CACHE
[ https://issues.apache.org/jira/browse/HIVE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778060#comment-13778060 ] Kousuke Saruta commented on HIVE-4501: -- I think we need to close FileSystem object properly, right? HS2 memory leak - FileSystem objects in FileSystem.CACHE Key: HIVE-4501 URL: https://issues.apache.org/jira/browse/HIVE-4501 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Critical Attachments: HIVE-4501.1.patch org.apache.hadoop.fs.FileSystem objects are getting accumulated in FileSystem.CACHE, with HS2 in unsecure mode. As a workaround, it is possible to set fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to true. Users should not have to bother with this extra configuration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Status: Open (was: Patch Available)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Attachment: (was: HIVE-5315.patch)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Attachment: HIVE-5315.patch
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Status: Patch Available (was: Open)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.patch
Sorry, I attached wrong patch yesterday. I've attached new patch.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used
[jira] [Commented] (HIVE-4501) HS2 memory leak - FileSystem objects in FileSystem.CACHE
[ https://issues.apache.org/jira/browse/HIVE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776963#comment-13776963 ] Kousuke Saruta commented on HIVE-4501: -- Hi [~thejas] Now, I try to address the issue HIVE-5296, which includes this issue. I think, it's better to call FileSystem.closeAll at the end of the session than disable cache because a FileSystem object can be used more than 1 time in a statement. HS2 memory leak - FileSystem objects in FileSystem.CACHE Key: HIVE-4501 URL: https://issues.apache.org/jira/browse/HIVE-4501 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4501.1.patch org.apache.hadoop.fs.FileSystem objects are getting accumulated in FileSystem.CACHE, with HS2 in unsecure mode. As a workaround, it is possible to set fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to false. Users should not have to bother with this extra configuration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4501) HS2 memory leak - FileSystem objects in FileSystem.CACHE
[ https://issues.apache.org/jira/browse/HIVE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777022#comment-13777022 ] Kousuke Saruta commented on HIVE-4501: -- Sorry, I think that we use FileSystem.closeAll may be not good idea because a thread can block others threads by synchronizing. So, [~thejas] idea may be better. HS2 memory leak - FileSystem objects in FileSystem.CACHE Key: HIVE-4501 URL: https://issues.apache.org/jira/browse/HIVE-4501 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4501.1.patch org.apache.hadoop.fs.FileSystem objects are getting accumulated in FileSystem.CACHE, with HS2 in unsecure mode. As a workaround, it is possible to set fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to false. Users should not have to bother with this extra configuration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777098#comment-13777098
]
Kousuke Saruta commented on HIVE-5296:
--
I found using FileSystem.closeAll is a bad idea and FIleSystem$Cache problem
will be addressed HIVE-4501 so I try to address another problem that opHandle
will not be released when Exception occurred during executing query or command.
I'll submit modified patch soon.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775564#comment-13775564
]
Kousuke Saruta commented on HIVE-5296:
--
I found the instance of Hashtable$Entry continuing to increase. This is caused
by two reasons as follows.
1. opHandle wouldn't release if Exception is thrown during executing query or
command
When Exception is thrown during executing query or command, operation handle
object will not be released from Map (OperationManager#handleToOperation)
because opHandleSet.add(opHandle) will not be executed in HiveSessionImpl
(Hiveserver2 side) and execResp.getOperationHandle() will not be executed in
HiveStatement (JDBC Client side).
{code}
public OperationHandle executeStatementInternal()
throws HiveSQLException {
acquire();
try {
ExecuteStatementOperation operation = getOperationManager()
.newExecuteStatementOperation(getSession(), statement, confOverlay,
runAsync);
opHandle
operation.run();
--- Throws exception and cannot get
handle.
OperationHandle opHandle = operation.getHandle();
opHandleSet.add(opHandle);
return opHandle;
} finally {
release();
}
}
{code}
{code}
public boolean execute(String sql) throws SQLException {
...
try {
closeClientOperation();
TExecuteStatementReq execReq = new TExecuteStatementReq(sessHandle, sql);
execReq.setConfOverlay(sessConf);
TExecuteStatementResp execResp = client.ExecuteStatement(execReq);
Utils.verifySuccessWithInfo(execResp.getStatus());
--- Throws exception and cannot get handle.
stmtHandle = execResp.getOperationHandle();
} catch (SQLException eS) {
throw eS;
} catch (Exception ex) {
throw new SQLException(ex.toString(), 08S01, ex);
}
...
{code}
2. FileSystem$Cache will be increase.
When we call FileSystem#get, FileSystem object is cached in FileSystem$Cache.
Cache is implemented using HashMap and equality of Key is implemented as
follows.
{code}
/** {@inheritDoc} */
public int hashCode() {
return (scheme + authority).hashCode() + ugi.hashCode() + (int)unique;
}
static boolean isEqual(Object a, Object b) {
return a == b || (a != null a.equals(b));
}
/** {@inheritDoc} */
public boolean equals(Object obj) {
if (obj == this) {
return true;
}
if (obj != null obj instanceof Key) {
Key that = (Key)obj;
return isEqual(this.scheme, that.scheme)
isEqual(this.authority, that.authority)
isEqual(this.ugi, that.ugi)
(this.unique == that.unique);
}
return false;
}
{code}
Key contains UserGroupInformation and two UserGroupInformation objects are
equivalent when the Subject objects included in each UserGroupInformation
object are same (not equivalent).
{code}
public boolean equals(Object o) {
if (o == this) {
return true;
} else if (o == null || getClass() != o.getClass()) {
return false;
} else {
return subject == ((UserGroupInformation) o).subject;
}
}
{code}
In Hiveserver2, it will get UserGroupInformation
UserGroupInformation#createRemoteUser or UserGroupInformation#createProxyUser.
Those methods create new Subject objects so Cache doesn't work.
If FileSystem.closeAll or FileSystem#close method are called, FileSystem object
will be removed from Cache.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.patch
I've create a patch of first idea.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar process is seen to
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Status: Patch Available (was: Open)
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar process is seen to increase extremely quickly, without
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775822#comment-13775822
]
Kousuke Saruta commented on HIVE-5296:
--
Hi, [~vgumashta] I've posted my patch to revoew board.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Affects Version/s: (was: 0.11.0)
0.13.0
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Fix Version/s: (was: 0.11.1)
0.13.0
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Status: Patch Available (was: Open)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Kousuke Saruta
Fix For: 0.13.0
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: (was: HIVE-5296.patch)
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar process is seen to increase extremely quickly, without
such space being released.
--
This
[jira] [Updated] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5296:
-
Attachment: HIVE-5296.patch
I've re-create a patch for HEAD.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Attachments: HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used by
the hiveserver2 runjar process is seen to
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Status: Open (was: Patch Available)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13772541#comment-13772541
]
Kousuke Saruta commented on HIVE-5296:
--
I found a cause of memory leak.
I will illustrate details and show an idea for modification later.
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
}
}
{code}
And by creating and closing many HiveClient objects. The heap space used
[jira] [Created] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION better way.
Kousuke Saruta created HIVE-5315:
Summary: bin/hive should retrieve HADOOP_VERSION better way.
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
In current implementation, bin/hive retrieve HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will not
be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Description:
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the first
line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will not
be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
was:
In current implementation, bin/hive retrieve HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will not
be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
bin/hive should retrieve HADOOP_VERSION better way.
---
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Attachment: HIVE-5315.patch
I've attached a patch for this issue.
bin/hive should retrieve HADOOP_VERSION better way.
---
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Status: Patch Available (was: Open)
bin/hive should retrieve HADOOP_VERSION better way.
---
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5315) bin/hive should retrieve HADOOP_VERSION by better way.
[
https://issues.apache.org/jira/browse/HIVE-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated HIVE-5315:
-
Summary: bin/hive should retrieve HADOOP_VERSION by better way. (was:
bin/hive should retrieve HADOOP_VERSION better way.)
bin/hive should retrieve HADOOP_VERSION by better way.
--
Key: HIVE-5315
URL: https://issues.apache.org/jira/browse/HIVE-5315
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Kousuke Saruta
Fix For: 0.11.1
Attachments: HIVE-5315.patch
In current implementation, bin/hive retrieves HADOOP_VERSION like as follows
{code}
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
{code}
But, sometimes, hadoop version doesn't show version information at the
first line.
If HADOOP_VERSION is not retrieve collectly, Hive or related processes will
not be up.
I faced this situation when I try to debug Hiveserver2 with debug option like
as follows
{code}
-Xdebug -Xrunjdwp:trunsport=dt_socket,suspend=n,server=y,address=9876
{code}
Then, hadoop version shows -Xdebug... at the first line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768919#comment-13768919
]
Kousuke Saruta commented on HIVE-5296:
--
I have some questions.
1. Which server-side (Hiveserver2 process) or client-side, does the memory leak
occur?
2. What query did you execute?
3. If you have already grasped, could you tell me which object increase?
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Labels: hiveserver
Fix For: 0.12.0
Original Estimate: 168h
Remaining Estimate: 168h
This error seems to relate to https://issues.apache.org/jira/browse/HIVE-3481
However, on inspection of the related patch and my built version of Hive
(patch carried forward to 0.12.0), I am still seeing the described behaviour.
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
logger.info(Problem closing connection +e);
}
}
[jira] [Commented] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[
https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731825#comment-13731825
]
Kousuke Saruta commented on HIVE-3363:
--
I think, this problem may be similar to HIVE-2137.
In SQLOperation.java, getNextRowSet() have a bunch of code as follows,
{code}
for (String rowString : rows) {
rowObj = serde.deserialize(new BytesWritable(rowString.getBytes()));
for (int i = 0; i fieldRefs.size(); i++) {
StructField fieldRef = fieldRefs.get(i);
fieldOI = fieldRef.getFieldObjectInspector();
deserializedFields[i] =
convertLazyToJava(soi.getStructFieldData(rowObj, fieldRef), fieldOI);
}
rowSet.addRow(resultSchema, deserializedFields);
}
{code}
The code above use getBytes() without setting encoding so it will use system
default encoding.
If the front end of hive is used in Windows, encoding mismatch will happen
because Hive(Hadoop) expects UTF-8 for their character encoding but Windows use
Shift_JIS.
So, I think the code above should be as follows
{code}
for (String rowString : rows) {
rowObj = serde.deserialize(new
BytesWritable(rowString.getBytes(UTF-8)));
for (int i = 0; i fieldRefs.size(); i++) {
StructField fieldRef = fieldRefs.get(i);
fieldOI = fieldRef.getFieldObjectInspector();
deserializedFields[i] =
convertLazyToJava(soi.getStructFieldData(rowObj, fieldRef), fieldOI);
}
rowSet.addRow(resultSchema, deserializedFields);
}
{code}
Special characters (such as 'é') displayed as '?' in Hive
-
Key: HIVE-3363
URL: https://issues.apache.org/jira/browse/HIVE-3363
Project: Hive
Issue Type: Bug
Reporter: Anand Balaraman
I am facing an issue while viewing special characters (such as é) using Hive.
If I view the file in HDFS (using hadoop fs -cat command), it is displayed
correctly as ’é’, but when I select the data using Hive, this character alone
gets replaced by a question mark.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[ https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-3363: - Attachment: HIVE-3363.patch Initial patch. Special characters (such as 'é') displayed as '?' in Hive - Key: HIVE-3363 URL: https://issues.apache.org/jira/browse/HIVE-3363 Project: Hive Issue Type: Bug Reporter: Anand Balaraman Attachments: HIVE-3363.patch I am facing an issue while viewing special characters (such as é) using Hive. If I view the file in HDFS (using hadoop fs -cat command), it is displayed correctly as ’é’, but when I select the data using Hive, this character alone gets replaced by a question mark. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[ https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-3363: - Affects Version/s: 0.12.0 Status: Patch Available (was: Open) Special characters (such as 'é') displayed as '?' in Hive - Key: HIVE-3363 URL: https://issues.apache.org/jira/browse/HIVE-3363 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Anand Balaraman Attachments: HIVE-3363.patch I am facing an issue while viewing special characters (such as é) using Hive. If I view the file in HDFS (using hadoop fs -cat command), it is displayed correctly as ’é’, but when I select the data using Hive, this character alone gets replaced by a question mark. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[ https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731592#comment-13731592 ] Kousuke Saruta commented on HIVE-3363: -- Hi Anand, Which OS are you using Hive? Special characters (such as 'é') displayed as '?' in Hive - Key: HIVE-3363 URL: https://issues.apache.org/jira/browse/HIVE-3363 Project: Hive Issue Type: Bug Reporter: Anand Balaraman I am facing an issue while viewing special characters (such as é) using Hive. If I view the file in HDFS (using hadoop fs -cat command), it is displayed correctly as ’é’, but when I select the data using Hive, this character alone gets replaced by a question mark. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[ https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731591#comment-13731591 ] Kousuke Saruta commented on HIVE-3363: -- Hi Anand, Which OS are you using Hive? Special characters (such as 'é') displayed as '?' in Hive - Key: HIVE-3363 URL: https://issues.apache.org/jira/browse/HIVE-3363 Project: Hive Issue Type: Bug Reporter: Anand Balaraman I am facing an issue while viewing special characters (such as é) using Hive. If I view the file in HDFS (using hadoop fs -cat command), it is displayed correctly as ’é’, but when I select the data using Hive, this character alone gets replaced by a question mark. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3363) Special characters (such as 'é') displayed as '?' in Hive
[ https://issues.apache.org/jira/browse/HIVE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731594#comment-13731594 ] Kousuke Saruta commented on HIVE-3363: -- Sorry for repeating submission... Special characters (such as 'é') displayed as '?' in Hive - Key: HIVE-3363 URL: https://issues.apache.org/jira/browse/HIVE-3363 Project: Hive Issue Type: Bug Reporter: Anand Balaraman I am facing an issue while viewing special characters (such as é) using Hive. If I view the file in HDFS (using hadoop fs -cat command), it is displayed correctly as ’é’, but when I select the data using Hive, this character alone gets replaced by a question mark. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Labels: patch (was: ) JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Labels: patch Fix For: 0.12.0 Attachments: HIVE-2137.patch, HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Attachment: HIVE-2137.patch I forgot to drop the table I added in tearDown. Thank you for your advice, tamtam180! JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Labels: patch Fix For: 0.12.0 Attachments: HIVE-2137.patch, HIVE-2137.patch, HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Attachment: HIVE-2137.patch Hi, I've re-fine the patch uploaded before. I've added a test code and test data. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch, HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13722789#comment-13722789 ] Kousuke Saruta commented on HIVE-2137: -- I wonder if my change really affects TestE2EScenarios.testReadOrcAndRCFromPig. I've just only added test code and test data for HiveQueryResultSet after the build successfully finished. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch, HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Description: JDBC driver decodes string by client side default encoding, which depends on operating system. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. was: JDBC driver decode string by client encoding. It ignore server encoding. For example, server = Linux (utf-8) client = Windows (shift-jis : it's japanese charset) It makes character corruption in the client. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch JDBC driver decodes string by client side default encoding, which depends on operating system. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Description: JDBC driver decodes string by client side default encoding, which depends on operating system unless we specify . It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. was: JDBC driver decodes string by client side default encoding, which depends on operating system. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch JDBC driver decodes string by client side default encoding, which depends on operating system unless we specify . It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Description: JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. was: JDBC driver decodes string by client side default encoding, which depends on operating system unless we specify . It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Description: JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. was: JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2137) JDBC driver doesn't encode string properly.
[ https://issues.apache.org/jira/browse/HIVE-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HIVE-2137: - Description: JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. was: JDBC driver decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. JDBC driver doesn't encode string properly. --- Key: HIVE-2137 URL: https://issues.apache.org/jira/browse/HIVE-2137 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.9.0 Reporter: Jin Adachi Fix For: 0.12.0 Attachments: HIVE-2137.patch JDBC driver for HiveServer1 decodes string by client side default encoding, which depends on operating system unless we don't specify another encoding. It ignore server side encoding. For example, when server side operating system and encoding are Linux (utf-8) and client side operating system and encoding are Windows (shift-jis : it's japanese charset, makes character corruption happens in the client. In current implementation of Hive, UTF-8 appears to be expected in server side so client side should encode/decode string as UTF-8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
