[jira] [Updated] (HIVE-2682) Clean-up logs
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2682: -- Attachment: HIVE-2682.D1035.1.patch rajat requested code review of HIVE-2682 [jira] Clean-up logs. Reviewers: JIRA Clean-up logs Just wanted to cleanup some logs being printed at wrong loglevel - 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000 rows for join key [...] as WARNING. Is it really that? 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed flushed and Begin Hash Table flush at close: size = 21 as WARNING. It shouldn't be. 4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning. Invalid statistic. which looks fishy. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1035 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2169/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Clean-up logs - Key: HIVE-2682 URL: https://issues.apache.org/jira/browse/HIVE-2682 Project: Hive Issue Type: Wish Components: Logging Reporter: Rajat Goel Priority: Trivial Attachments: HIVE-2682.D1035.1.patch Original Estimate: 24h Remaining Estimate: 24h Just wanted to cleanup some logs being printed at wrong loglevel - 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000 rows for join key [...] as WARNING. Is it really that? 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed flushed and Begin Hash Table flush at close: size = 21 as WARNING. It shouldn't be. 4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning. Invalid statistic. which looks fishy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2682) Clean-up logs
Clean-up logs
-
Key: HIVE-2682
URL: https://issues.apache.org/jira/browse/HIVE-2682
Project: Hive
Issue Type: Wish
Components: Logging
Reporter: Rajat Goel
Priority: Trivial
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000
rows for join key [...] as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed
flushed and Begin Hash Table flush at close: size = 21 as WARNING. It
shouldn't be.
3. org.apache.hadoop.fs.FileSystem prints Not initializing standby filesystem
because the needed configuration parameters fs.default.name{0|1} are missing.
as WARNING. Doesn't look like a problem since it is happening at very high rate.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning.
Invalid statistic. which looks fishy.
5. org.apache.hadoop.mapred.Merger prints totalBytesProcessed: ABC is larger
than totalBytesExpected: XYZ as ERROR at high rate...do we actually care about
this??
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.23.0 - Build # 50 - Still Failing
Changes for Build #1 Changes for Build #2 [jvs] HIVE-2625. Fix maven-build Ant target (Carl Steinbach via jvs) Changes for Build #3 Changes for Build #4 [jvs] HIVE-2630 [jira] TestHiveServer doesn't produce a JUnit report file (Marek Sapota via John Sichi) Summary: Remove property overrides Run `ant test delDtestcase=TestHiveServer ant testreport`, the test report will be empty because not 'TEST/del*.xml' file was produced by JUnit. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 621 [jvs] HIVE-2362. HiveConf properties not appearing in the output of 'set' or 'set -v' (Carl Steinbach via jvs) Changes for Build #5 [hashutosh] HIVE-2467 : HA Support for Metastore Server (Thomas Weise via Ashutosh Chauhan) [hashutosh] HIVE-2622 : Hive POMs reference the wrong Hadoop artifacts (Carl Steinbach via Ashutosh Chauhan) Changes for Build #6 Changes for Build #7 [heyongqiang] HIVE-1003 [jira] optimize metadata only queries (Namit Jain via Yongqiang He) Summary: testing Queries like: select max(ds) from T where ds is a partitioning column should be optimized. Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: njain, heyongqiang Differential Revision: 105 Changes for Build #8 Changes for Build #9 [heyongqiang] HIVE-2634 [jira] revert HIVE-2566 (Namit Jain via Yongqiang He) Summary: HIVE-2634 This is leading to some problems. I will upload the offending testcase in a new jira. Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: 651 Changes for Build #10 [heyongqiang] HIVE-2632 [jira] ignore exception for external jars via reflection (Namit Jain via Yongqiang He) Summary: HIVE-2632 Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: 639 Changes for Build #11 Changes for Build #12 [heyongqiang] HIVE-2635 [jira] wrong class loader used for external jars (Namit Jain via Yongqiang He) Summary: HIVE-2635 Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: 687 [namit] HIVE-2329 Not using map aggregation, fails to execute group-by after cluster-by with same key (Navis via namit) Changes for Build #13 [namit] HIVE-2628 move one line log from MapOperator to HiveContextAwareRecordReader (He Yongqiang via namit) [namit] HIVE-2520 left semi join will duplicate data (binlijin via namit) [jvs] HIVE-1487 [jira] Add ptest script (Marek Sapota via John Sichi) Summary: HIVE-1464 speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 603 Changes for Build #14 Changes for Build #15 Changes for Build #16 Changes for Build #17 [jvs] Fix executable permissions left out of commit for HIVE-1487. Changes for Build #18 [jvs] HIVE-2647 [jira] Force Bash shell on parallel test slave nodes (Marek Sapota via John Sichi) Summary: Forcing Bash on each remote command call. Force use of Bash shell, so user default shell choice doesn't impact the test script. Test Plan: EMPTY Reviewers: JIRA, jsichi, njain Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 771 [jvs] HIVE-2648 [jira] Parallel tests fail if master directory is not present (Marek Sapota via John Sichi) Summary: Fix the task downloading Ant. Parallel tests should create directories as needed. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 777 Changes for Build #19 [jvs] HIVE-2649 [jira] Allow multiple ptest runs by the same person (Marek Sapota via John Sichi) Summary: Allow multiple ptest runs. Allow running ptest simultaneously, multiple times by one person on the same machines. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 783 [jvs] HIVE-2650 [jira] Parallel test commands that include cd fail (Marek Sapota via John Sichi) Summary: Add quotes around bash -c to fix commands using cd. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 801 [jvs] HIVE-2643. Recent patch prevents Hadoop confs from loading in 0.20.204 (Carl Steinbach via jvs) Changes for Build #20 Changes for Build #21 [jvs] HIVE-1040 [jira] use sed rather than diff for masking out noise in diff-based tests (Marek Sapota via John Sichi) Summary: Replace diff -I with regex masking in Java The current diff -I approach has two problems: (1) it does not allow resolution finer than line-level, so it's impossible to mask out pattern
[jira] [Updated] (HIVE-2682) Clean-up logs
[
https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajat Goel updated HIVE-2682:
-
Description:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000
rows for join key [...] as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed
flushed and Begin Hash Table flush at close: size = 21 as WARNING. It
shouldn't be.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning.
Invalid statistic. which looks fishy.
was:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000
rows for join key [...] as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed
flushed and Begin Hash Table flush at close: size = 21 as WARNING. It
shouldn't be.
3. org.apache.hadoop.fs.FileSystem prints Not initializing standby filesystem
because the needed configuration parameters fs.default.name{0|1} are missing.
as WARNING. Doesn't look like a problem since it is happening at very high rate.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning.
Invalid statistic. which looks fishy.
5. org.apache.hadoop.mapred.Merger prints totalBytesProcessed: ABC is larger
than totalBytesExpected: XYZ as ERROR at high rate...do we actually care about
this??
Clean-up logs
-
Key: HIVE-2682
URL: https://issues.apache.org/jira/browse/HIVE-2682
Project: Hive
Issue Type: Wish
Components: Logging
Reporter: Rajat Goel
Priority: Trivial
Original Estimate: 24h
Remaining Estimate: 24h
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000
rows for join key [...] as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table
completed flushed and Begin Hash Table flush at close: size = 21 as
WARNING. It shouldn't be.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning.
Invalid statistic. which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2675) JDBC SQL execution exception does not contain cause
[
https://issues.apache.org/jira/browse/HIVE-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176264#comment-13176264
]
Istvan Szegedi commented on HIVE-2675:
--
Could you shed some lights what release exactly you are talking about? In hive
0.8.0 the HiveStatement.java class seems to have executeQuery method, not
executeSQL.
(./hive/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java)
Anyway, the toString() method of the Exception - which is actually inherited
from Throwable class - returns the name of the object and the message, as the
result of the getMessage() method, concatenated by a colon. Thus when the
SQLException is rethrown from the second catch block, the SQLException will
have the original message and also the SQLState string value- 08S01.
See for instance, a simple code to demonstrate it:
import java.sql.SQLException;
import java.io.FileReader;
public class Exc {
public static void main(String[] args) {
try {
ThrowException();
}
catch (SQLException ex) {
System.out.println(String: + ex.toString());
System.out.println(State: + ex.getSQLState());
System.out.println(Message: + ex.getMessage());
System.exit(1);
}
}
public static void ThrowException() throws SQLException {
try {
FileReader fr = new FileReader(test.txt);
}
catch (Exception ex) {
System.out.println(FileReader exception caught: + ex.toString());
throw new SQLException(ex.toString(), 08S01);
}
}
}
The output of this code is as follows (the original FileNotFoundException is
returned by the rethrown exception and the SQLState can be retrieved, too):
FileReader exception caught: java.io.FileNotFoundException: test.txt (No such
file or directory)
String: java.sql.SQLException: java.io.FileNotFoundException: test.txt (No such
file or directory)
State: 08S01
Message: java.io.FileNotFoundException: test.txt (No such file or directory)
JDBC SQL execution exception does not contain cause
---
Key: HIVE-2675
URL: https://issues.apache.org/jira/browse/HIVE-2675
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.8.0
Environment: Any
Reporter: Greg Cottman
If SQL execution throws an exception in the HiveStatement.executeSQL() method
then it's message is rethrown as a SQLException with a SQLState of 08S01:
try {
resultSet = null;
client.execute(sql);
} catch (HiveServerException e) {
throw new SQLException(e.getMessage(), e.getSQLState(), e.getErrorCode());
} catch (Exception ex) {
throw new SQLException(ex.toString(), 08S01);
}
In the case of failed DDL, the exception ex has a cause - such as a
java.io.IOException - that contains the actual error text. The description
of the actual problem is lost by failing to include ex as the cause in the
new SQLException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176269#comment-13176269 ] Phabricator commented on HIVE-2682: --- raghotham has commented on the revision HIVE-2682 [jira] Clean-up logs. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java:128 you dont really need to say 'Warning' - it will show up in the log anyway. ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java:112-114 what does this comment even mean? :) REVISION DETAIL https://reviews.facebook.net/D1035 Clean-up logs - Key: HIVE-2682 URL: https://issues.apache.org/jira/browse/HIVE-2682 Project: Hive Issue Type: Wish Components: Logging Reporter: Rajat Goel Priority: Trivial Attachments: HIVE-2682.D1035.1.patch Original Estimate: 24h Remaining Estimate: 24h Just wanted to cleanup some logs being printed at wrong loglevel - 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000 rows for join key [...] as WARNING. Is it really that? 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed flushed and Begin Hash Table flush at close: size = 21 as WARNING. It shouldn't be. 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning. Invalid statistic. which looks fishy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1172 - Still Failing
Changes for Build #1144 [jvs] HIVE-1040 [jira] use sed rather than diff for masking out noise in diff-based tests (Marek Sapota via John Sichi) Summary: Replace diff -I with regex masking in Java The current diff -I approach has two problems: (1) it does not allow resolution finer than line-level, so it's impossible to mask out pattern occurrences within a line, and (2) it produces unmasked files, so if you run diff on the command line to compare the result .q.out with the checked-in file, you see the noise. My suggestion is to first run sed to replace noise patterns with an unlikely-to-occur string like ZYZZYZVA, and then diff the pre-masked files without using any -I. This would require a one-time hit to update all existing .q.out files so that they would contain the pre-masked results. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: jsichi Differential Revision: 597 Changes for Build #1145 Changes for Build #1146 [namit] HIVE-2640 Add alterPartition to AlterHandler interface (Kevin Wilfong via namit) Changes for Build #1147 [namit] HIVE-2617 Insert overwrite table db.tname fails if partition already exists (Chinna Rao Lalam via namit) Changes for Build #1148 [heyongqiang] HIVE-2651 [jira] The variable hive.exec.mode.local.auto.tasks.max should be changed (Namit Jain via Yongqiang He) Summary: HIVE-2651 It should be called hive.exec.mode.local.auto.input.files.max instead. The number of input files are checked currently. Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: 861 [cws] HIVE-727. Hive Server getSchema() returns wrong schema for 'Explain' queries (Prasad Mujumdar via cws) [namit] HIVE-2611 Make index table output of create index command if index is table based (Kevin Wilfong via namit) Changes for Build #1150 [jvs] HIVE-2657 [jira] builtins JAR is not being published to Maven repo hive-cli POM does not depend on it either (Carl Steinbach via John Sichi) Summary: Make hive-cli and hive-ql depend on hive-builtins Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: jsichi Differential Revision: 897 [namit] HIVE-2654 hive.querylog.location requires parent directory to be exist or else folder creation fails (Chinna Rao Lalam via namit) Changes for Build #1151 [hashutosh] HIVE-1892 : show functions also returns internal operators (Priyadarshini via Ashutosh Chauhan) Changes for Build #1152 Changes for Build #1153 [namit] HIVE-2660 Need better exception handling in RCFile tolerate corruptions mode (Ramkumar Vadali via namit) Changes for Build #1154 [cws] HIVE-2631. Make Hive work with Hadoop 1.0.0 (Ashutosh Chauhan via cws) Changes for Build #1155 [cws] HIVE-BUILD. Update RELEASE_NOTES.txt with 0.8.0 release information (cws) Changes for Build #1156 Changes for Build #1157 Changes for Build #1158 [namit] HIVE-2602 add support for insert partition overwrite(...) if not exists (Chinna Rao Lalam via namit) Changes for Build #1159 Changes for Build #1160 [cws] HIVE-2005. Implement BETWEEN operator (Navis via cws) Changes for Build #1161 [jvs] HIVE-2433. add DOAP file for Hive Changes for Build #1162 Changes for Build #1163 Changes for Build #1164 [heyongqiang] HIVE-2666 [jira] StackOverflowError when using custom UDF in map join (Kevin Wilfong via Yongqiang He) Summary: Resource files are now added to the class path as soon as they are added via the CLI. This fixes the stack overflow error mentioned in the JIRA by ensuring a consistent class loader between serializers and deserializers for the same query. Note that now serdes which contain a static block to register themselves are now registered twice, once when adding the file to the class loader, and once when an instance of the class is created. Previously, registering a serde twice resulted in an exception, to avoid this, I have downgraded it to a warning. When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error. Test Plan: I ran the unit tests to verify nothing was broken. I ran several queries which used custom UDFs and involved a join which was converted to a map join. I verified these completed successfully consistently Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang, kevinwilfong Differential Revision: 957 [namit] HIVE-2642 fix Hive-2566 and make union optimization more aggressive (Yongqiang He via namit) Changes for Build #1166 Changes for Build #1167 Changes for Build #1168 [heyongqiang] HIVE-2600: Enable/Add type-specific compression for rcfile (Krishna Kumar via He Yongqiang) Changes for Build #1169 Changes for Build #1170 [cws] HIVE-1877. Add java_method() as a synonym for the reflect() UDF (Zhenxiao Luo via cws) Changes for Build
[jira] [Created] (HIVE-2683) load data twice from the same hdfs file without overwrite fails with return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
load data twice from the same hdfs file without overwrite fails with return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Key: HIVE-2683 URL: https://issues.apache.org/jira/browse/HIVE-2683 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.7.1 Environment: Cloudera VM for Linux Red Hat Reporter: Bobo Vava I try to load data into a hive table twice from the same hdfs file without overwrite option. I would expect second load to append to existing data or at least replace it. I get the following error message instead. Note that, during the operation Hive is renaming the source file to original_name_copy_nbr, which makes us believe it is trying to avoid erasing the existing file while moving and both files (original and renamed) would be made available to Hive. But this is not the case. Example of code end error stack : load data inpath 'hdfs://0.0.0.0/user/qafiles/SRC_FILE_EMP.dat' into table EMP_DELIMITED_FILE_TRUNC_TBL --- Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask river returned: 9. Errors: Hive history file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt Loading data to table default.emp_delimited_file_trunc_tbl Failed with exception null FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Failed with exception null 11/12/13 22:13:55 ERROR exec.MoveTask: Failed with exception null java.lang.NullPointerException at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1738) at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:542) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1189) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:197) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:306) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxService Impl.java:499) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java :886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 ) at java.lang.Thread.run(Thread.java:619) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask 11/12/13 22:13:55 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask 11/12/13 22:13:55 ERROR beeswax.BeeswaxServiceImpl: Exception while processing query BeeswaxException(message:Driver returned: 9. Errors: Hive history file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt Loading data to table default.emp_delimited_file_trunc_tbl Failed with exception null FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask , log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f, handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f, log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f)) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110) at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at
[jira] [Commented] (HIVE-2681) SUCESS is misspelled
[ https://issues.apache.org/jira/browse/HIVE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176354#comment-13176354 ] Ashutosh Chauhan commented on HIVE-2681: +1 .Running tests. In the meanwhile can you upload the patch on jira and grant ASF perms. SUCESS is misspelled Key: HIVE-2681 URL: https://issues.apache.org/jira/browse/HIVE-2681 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2681.D1029.1.patch C'mon! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2684) Provide compile + run method for HiveServer interface
Provide compile + run method for HiveServer interface - Key: HIVE-2684 URL: https://issues.apache.org/jira/browse/HIVE-2684 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Navis Priority: Minor Fix For: 0.9.0 In our use cases, we need to access query plan in early time (before execution). 1. compile 2. get query plan 3. setup configuration (callback, etc.) 4. run the query Additional methods are needed for 1, 3, 4 stage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2685) Unable to query Hive table defined with custom ROW FORMAT remotely using JDBC
Unable to query Hive table defined with custom ROW FORMAT remotely using JDBC
-
Key: HIVE-2685
URL: https://issues.apache.org/jira/browse/HIVE-2685
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.7.1
Environment: Cloudera VM for Linux Red Hat
Reporter: Bobo Vava
Created table using custom format as follows :
create table if not exists FILE_HDFS_TO_HIVE_CUSTOM_NO_STAGE_TGT
(
ID STRING,
NAMESTRING,
DESCR STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
input.regex = (.{3})(.{5})(.{9}),
output.format.string = %1$s %2$s %3$s
)
STORED AS TEXTFILE
and loaded some data into it.
Now I am trying to query the table from a remote host using JDBC from a java
program.
All the Hive JDBC jars are included in the classpath and I am able to execute
run all kinds
of DDL and DML statements except for this one.
Additionally I use add jar just before executing the query :
add jar
/ade/bosimova_bobo_odi/odi/odiqa/automation/qa_scripts/knowledge_modules/sql/hive_cert/drivers/hive-contrib-0.7.1-cdh3u2.jar
0
select * from file_hdfs_to_hive_custom_no_stage_tgt order by id
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2
from org.apache.hadoop.hive.ql.exec.MapRedTask
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.contrib.serde2.RegexSerDe
More details :
2011-12-27 19:59:30,695 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.contrib.serde2.RegexSerDe
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:403)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
... 22 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.contrib.serde2.RegexSerDe
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:228)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:359)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please
[jira] [Updated] (HIVE-2681) SUCESS is misspelled
[ https://issues.apache.org/jira/browse/HIVE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Chang updated HIVE-2681: - Attachment: HIVE-2681.D1029.1.patch SUCESS is misspelled Key: HIVE-2681 URL: https://issues.apache.org/jira/browse/HIVE-2681 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2681.D1029.1.patch, HIVE-2681.D1029.1.patch C'mon! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176429#comment-13176429 ] Phabricator commented on HIVE-2682: --- rajat has commented on the revision HIVE-2682 [jira] Clean-up logs. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java:128 :P Then I need to change this in multiple places. ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java:112-114 I think this is related to skew joins. If we touch 1000 rows for a single key, there is a chance of data being skewed for that key and it needs to be handled specially. (I am just guessing, though). REVISION DETAIL https://reviews.facebook.net/D1035 Clean-up logs - Key: HIVE-2682 URL: https://issues.apache.org/jira/browse/HIVE-2682 Project: Hive Issue Type: Wish Components: Logging Reporter: Rajat Goel Priority: Trivial Attachments: HIVE-2682.D1035.1.patch Original Estimate: 24h Remaining Estimate: 24h Just wanted to cleanup some logs being printed at wrong loglevel - 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000 rows for join key [...] as WARNING. Is it really that? 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed flushed and Begin Hash Table flush at close: size = 21 as WARNING. It shouldn't be. 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning. Invalid statistic. which looks fishy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2684) Provide compile + run method for HiveServer interface
[ https://issues.apache.org/jira/browse/HIVE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2684: -- Attachment: HIVE-2684.D1041.1.patch navis requested code review of HIVE-2684 [jira] Provide compile + run method for HiveServer interface. Reviewers: JIRA DPAL-165 Split thrift_client#execute to compile + run for early access to execution plan In our use cases, we need to access query plan in early time (before execution). 1. compile 2. get query plan 3. setup configuration (callback, etc.) 4. run the query Additional methods are needed for 1, 3, 4 stage. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1041 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java service/if/hive_service.thrift service/src/gen/thrift/gen-cpp/ThriftHive.cpp service/src/gen/thrift/gen-cpp/ThriftHive.h service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java service/src/gen/thrift/gen-php/hive_service/ThriftHive.php service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote service/src/gen/thrift/gen-py/hive_service/ThriftHive.py service/src/gen/thrift/gen-rb/thrift_hive.rb service/src/java/org/apache/hadoop/hive/service/HiveServer.java service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2175/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Provide compile + run method for HiveServer interface - Key: HIVE-2684 URL: https://issues.apache.org/jira/browse/HIVE-2684 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Navis Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2684.D1041.1.patch In our use cases, we need to access query plan in early time (before execution). 1. compile 2. get query plan 3. setup configuration (callback, etc.) 4. run the query Additional methods are needed for 1, 3, 4 stage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2682: -- Attachment: HIVE-2682.D1035.2.patch rajat updated the revision HIVE-2682 [jira] Clean-up logs. Reviewers: JIRA, jsichi, jonchang, heyongqiang, njain Took care of the comments REVISION DETAIL https://reviews.facebook.net/D1035 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java Clean-up logs - Key: HIVE-2682 URL: https://issues.apache.org/jira/browse/HIVE-2682 Project: Hive Issue Type: Wish Components: Logging Reporter: Rajat Goel Priority: Trivial Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch Original Estimate: 24h Remaining Estimate: 24h Just wanted to cleanup some logs being printed at wrong loglevel - 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints table 0 has 1000 rows for join key [...] as WARNING. Is it really that? 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints Hash Table completed flushed and Begin Hash Table flush at close: size = 21 as WARNING. It shouldn't be. 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints Warning. Invalid statistic. which looks fishy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2684) Provide compile + run method for HiveServer interface
[ https://issues.apache.org/jira/browse/HIVE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2684: -- Attachment: HIVE-2684.D1041.2.patch navis updated the revision HIVE-2684 [jira] Provide compile + run method for HiveServer interface. Reviewers: JIRA revert filemode of 'service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote' REVISION DETAIL https://reviews.facebook.net/D1041 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java service/if/hive_service.thrift service/src/gen/thrift/gen-cpp/ThriftHive.cpp service/src/gen/thrift/gen-cpp/ThriftHive.h service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java service/src/gen/thrift/gen-php/hive_service/ThriftHive.php service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote service/src/gen/thrift/gen-py/hive_service/ThriftHive.py service/src/gen/thrift/gen-rb/thrift_hive.rb service/src/java/org/apache/hadoop/hive/service/HiveServer.java service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java Provide compile + run method for HiveServer interface - Key: HIVE-2684 URL: https://issues.apache.org/jira/browse/HIVE-2684 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Navis Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2684.D1041.1.patch, HIVE-2684.D1041.2.patch In our use cases, we need to access query plan in early time (before execution). 1. compile 2. get query plan 3. setup configuration (callback, etc.) 4. run the query Additional methods are needed for 1, 3, 4 stage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2686) Test ppr_pushdown.q is failing on trunk
Test ppr_pushdown.q is failing on trunk
---
Key: HIVE-2686
URL: https://issues.apache.org/jira/browse/HIVE-2686
Project: Hive
Issue Type: Test
Components: Testing Infrastructure
Affects Versions: 0.9.0
Reporter: Ashutosh Chauhan
Fix For: 0.9.0
{noformat}
$ ant test -Dtestcase=TestCliDriver -Dqfile=ppr_pushdown.q
[junit] See build/ql/tmp/hive.log, or try ant test ...
-Dtest.silent=false to get more logs.
[junit] at junit.framework.Assert.fail(Assert.java:50)
[junit] at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown(TestCliDriver.java:129)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:168)
[junit] at junit.framework.TestCase.runBare(TestCase.java:134)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
[junit] at junit.framework.TestResult.run(TestResult.java:113)
[junit] at junit.framework.TestCase.run(TestCase.java:124)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
[junit] at junit.framework.TestSuite.run(TestSuite.java:238)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
[junit] 178d177
[junit] 1234 1234
[junit] 179a179
[junit] 1234 1234
[junit] 196d195
[junit] 1224 1224
[junit] 197a197
[junit] 1224 1224
[junit] 214d213
[junit] Exception: Client execution results failed with error code = 1
[junit] See build/ql/tmp/hive.log, or try ant test ...
-Dtest.silent=false to get more logs.
[junit] 1214 1214
[junit] 215a215
[junit] 1214 1214
[junit] 232d231
[junit] 12.4 12.4
[junit] 233a233
[junit] 12.4 12.4
[junit] 250d249
[junit] 12+4 12+4
[junit] 251a251
[junit] 12+4 12+4
[junit] 268d267
[junit] 12:4 12:4
[junit] 269a269
[junit] 12:4 12:4
[junit] 286d285
[junit] 12%4 12%4
[junit] 287a287
[junit] 12%4 12%4
[junit] 304d303
[junit] 12*4 12*4
[junit] 305a305
[junit] 12*4 12*4
[junit] Cleaning up TestCliDriver
[junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 137.556 sec
[junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2681) SUCESS is misspelled
[ https://issues.apache.org/jira/browse/HIVE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176472#comment-13176472 ] Ashutosh Chauhan commented on HIVE-2681: HIVE-2686 is currently failing on trunk. SUCESS is misspelled Key: HIVE-2681 URL: https://issues.apache.org/jira/browse/HIVE-2681 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2681.D1029.1.patch, HIVE-2681.D1029.1.patch C'mon! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
