[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200337#comment-14200337
]
Brock Noland commented on HIVE-8744:
works for me!
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(Unknown Source)
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200561#comment-14200561
]
Hive QA commented on HIVE-8744:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12679872/HIVE-8744.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6674 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1667/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1667/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12679872 - PreCommit-HIVE-TRUNK-Build
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch, HIVE-8744.2.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200659#comment-14200659
]
Szehon Ho commented on HIVE-8744:
-
+1, thanks.
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch, HIVE-8744.2.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(Unknown
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200683#comment-14200683
]
Prasanth J commented on HIVE-8744:
--
HIVE-8735 is also addressing the same problem. Usually the client which
publishes provides the key (FSOperator, StatsTask) has some logic to trim down
the length of the key using MD5 hash. If the key gets greater than max stats
key prefix (from hive config), Utilities.getHashedPrefixKey() method is invoked
to get a smaller length key. Can you try with the patch from HIVE-8735 to see
if the test case works? HIVE-8735 truncates the key before publishing.
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch, HIVE-8744.2.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200933#comment-14200933
]
Sergio Peña commented on HIVE-8744:
---
That patch works well [~prasanth_j].
We can use the one from HIVE-8735 instead.
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch, HIVE-8744.2.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199439#comment-14199439
]
Szehon Ho commented on HIVE-8744:
-
Is it the right patch? It looks like this is adding dependency to distcp, so
how will it solve the issue?
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: CDH-23009.1.java
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199451#comment-14199451
]
Sergio Peña commented on HIVE-8744:
---
Thanks [~szehon].
I submit the correct patch.
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199462#comment-14199462
]
Szehon Ho commented on HIVE-8744:
-
Thanks Sergio. It looks ok to me, only comment is that old tables need to be
re-created, which might warrant a release note.
Also, some old version of mysql before 5.0.3 doesn't seem to support varchar
beyond 255, not sure if its a concern. [~brocknoland] do you happen to know?
Thanks
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199770#comment-14199770
]
Brock Noland commented on HIVE-8744:
That's a pretty old database and will be older when we release 0.15. I think we
should move ahead...
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink
VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to
length 255.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199887#comment-14199887
]
Szehon Ho commented on HIVE-8744:
-
Sounds good. Another thought came to mind, what do you guys think about rename
the table to v3? I saw it was done for a schema change in HIVE-2471 awhile
back.
That way, users dont have to manually drop the table/schema, and we can just in
release notes doc to say that you can delete v2. Just wanted to bring it up,
not sure what you guys think. Thanks.
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: A truncation error was encountered trying
to shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 30
[jira] [Commented] (HIVE-8744) hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large
[
https://issues.apache.org/jira/browse/HIVE-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199956#comment-14199956
]
Hive QA commented on HIVE-8744:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12679708/HIVE-8744.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6674 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1657/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1657/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1657/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12679708 - PreCommit-HIVE-TRUNK-Build
hbase_stats3.q test fails when paths stored at
JDBCStatsUtils.getIdColumnName() are too large
-
Key: HIVE-8744
URL: https://issues.apache.org/jira/browse/HIVE-8744
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-8744.1.patch
This test is related to the bug HIVE-8065 where I am trying to support HDFS
encryption. One of the enhancements to support it is to create a
.hive-staging directory on the same table directory location where the query
is executed.
Now, when running the hbase_stats3.q test from a temporary directory that has
a large path, then the new path, a combination of table location +
.hive-staging + random temporary subdirectories, is too large to fit into the
statistics table, so the path is truncated.
This causes the following error:
{noformat}
2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]:
jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error
during publishing statistics.
java.sql.SQLDataException: A truncation error was encountered trying to
shrink VARCHAR
'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.' to length 255.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
at
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
