[jira] [Created] (HIVE-24181) Hive bug prevents allowed characters in nested column
Jainik Vora created HIVE-24181:
--
Summary: Hive bug prevents allowed characters in nested column
Key: HIVE-24181
URL: https://issues.apache.org/jira/browse/HIVE-24181
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.3.5
Reporter: Jainik Vora
Hive allows any unicode character in column name when specified within backtick
(`). But this behavior didn’t work for a nested column containing hyphen (-)
which points to a possible bug in hive DDL parsing logic.
_Special characters at first level column works fine_
{code:java}
CREATE EXTERNAL TABLE test_dwh.sample_parquet_table (
`user-agent`: string, -- first level column with “-” works
`context` struct<
`os`: struct<
`name`: string,
`version`: string
>
>
) PARTITIONED BY (day string)
STORED as PARQUET
LOCATION 's3://xxx/tmp/sample_data/events'
{code}
_Special characters in nested column throws error_
{code:java}
CREATE EXTERNAL TABLE test_dwh.sample_parquet_table (
`context` struct<
`os`: struct<
`name`: string,
`version`: string,
`is-test`: string -- nested column with “-” throws error
>
>
) PARTITIONED BY (day string)
STORED as PARQUET
LOCATION 's3://xxx/tmp/sample_data/events' {code}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException:
Error: : expected at the position 46 of
'struct>' but '-' is found.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24180) 'hive.txn.heartbeat.threadpool.size' is deprecated in HiveConf with no alternative
Stamatis Zampetakis created HIVE-24180: -- Summary: 'hive.txn.heartbeat.threadpool.size' is deprecated in HiveConf with no alternative Key: HIVE-24180 URL: https://issues.apache.org/jira/browse/HIVE-24180 Project: Hive Issue Type: Bug Reporter: Stamatis Zampetakis Assignee: Stamatis Zampetakis HiveConf.ConfVars#HIVE_TXN_HEARTBEAT_THREADPOOL_SIZE appears deprecated with javadoc pointing to MetastoreConf.TXN_HEARTBEAT_THREADPOOL_SIZE but there is no such configuration variable in MetastoreConf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
Stamatis Zampetakis created HIVE-24179:
--
Summary: Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS
statement
Key: HIVE-24179
URL: https://issues.apache.org/jira/browse/HIVE-24179
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
Fix For: 4.0.0
Attachments: summary.png
The problem can be reproduced by executing repeatedly a SHOW LOCK statement and
monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a
few minutes before the server crashes with OutOfMemory error such as the one
shown below.
{noformat}
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
at
org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
at
org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
at
org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
at
org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
at
org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
at
org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
at
org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
at
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
at
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
{noformat}
The heap dump shows (summary.png) that most of the memory is consumed by
{{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive
configurations referenced by {{DbTxnManager}}.
The latter are not eligible for garbage collection since at
[construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212]
time they are passed implicitly in a callback stored inside
ShutdownHookManager.
When the {{DbTxnManager}} is closed properly the leak is not present since the
callback is
[removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882]
from ShutdownHookManager.
{{SHOW LOCKS}} statements create
([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52],
What is the latest version of JDK supported by hive 3.1.2?
Hello: What is the latest version of JDK supported by hive 3.1.2? thinks. I am looking forward to your reply??
[jira] [Created] (HIVE-24178) managedlocation is missing in SHOW CREATE DATABASE
Csaba Ringhofer created HIVE-24178: -- Summary: managedlocation is missing in SHOW CREATE DATABASE Key: HIVE-24178 URL: https://issues.apache.org/jira/browse/HIVE-24178 Project: Hive Issue Type: Bug Reporter: Csaba Ringhofer The output of SHOW CREATE DATABASE contains location, but doesn't contain managed location, so the database it would create would be actually different. To reproduce: create database db1 location "/test-warehouse/a" managedlocation "test-warehouse/b"; show create database db1; result: +--+ |createdb_stmt | +--+ | CREATE DATABASE `db1`| | LOCATION | | 'hdfs://localhost:20500/test-warehouse/a' | +--+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24177) hive mapjoin udf class not found
xiaohang.li created HIVE-24177: -- Summary: hive mapjoin udf class not found Key: HIVE-24177 URL: https://issues.apache.org/jira/browse/HIVE-24177 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 2.3.6, 2.2.0 Reporter: xiaohang.li when i use mapjoin,hive throws Execution failed with exit status. 以下是我的测试demo: CREATE TABLE `fact`(CREATE TABLE `fact`( `id` int, `name` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='\t', 'serialization.format'='\t') ; vi fact.txt1 lee2 lee2 load data local inpath 'fact.txt' into table fact; create table dim_table( id int, name string)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='\t', 'serialization.format'='\t'); vi dim.txt1 lee2 test3 test34 test45 test56 test67 test8 test9 test10 test load data local inpath 'dim.txt' into table dim_table; then i create a lowwer udf : add jar udfdemo-1.0.jar; create temporary function mylower as 'com.test.hive.Lower'; select * from( select id,name,mylower1(name) user_id from fact )t1 join dim_table t2 on t2.name='test'; the error is below: 2020-09-18 14:39:16 Starting to launch local task to process map join; maximum memory = 149133721602020-09-18 14:39:16 Starting to launch local task to process map join; maximum memory = 14913372160Execution failed with exit status: 2Obtaining error information Task failed!Task ID: Stage-4 log4j is thows below exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.test.hive.Lower -- This message was sent by Atlassian Jira (v8.3.4#803005)
