[jira] [Commented] (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010099#comment-13010099 ] Bennie Schut commented on HIVE-1675: I'm unable to reproduce this on the current trunk. I still get errors on local mode but it's something different java.lang.IllegalStateException(Shutdown in progress) after witch the server shuts down. I'll make a new jira for that and I think we can close this one. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010103#comment-13010103 ] Bennie Schut commented on HIVE-1675: Added HIVE-2073 for the different error we now get on local mode. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12970201#action_12970201 ] Joydeep Sen Sarma commented on HIVE-1675: - the combination of auto local and parallel had a bug that was recently fixed (HIVE-1776) The stack reported above is weird - we should not be entering this code path. as u mentioned - HIVE-1846 will fix this - but currently, the getMapRedWork call invoked from different places in the stack should be satisfied from the an in-memory cache (and shouldn't hit the file). the file is not written out for local mode (because there's an assumption that everything runs in the same jvm). Unable to explain this - i think this is worth fixing because the plan file processing takes some time and it's better to retrieve it from memory where possible. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968759#action_12968759 ] Bennie Schut commented on HIVE-1675: manage to cause this with parallel=false so perhaps not such an interesting angle ;-) I've added some more logging to better understand the cause of this: 2010-12-07 15:49:44,697 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:9c2eeba4-a602-4d4b-ba0b-60ce815c4ea7 from cache. 2010-12-07 15:49:44,703 INFO lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:clinit(34)) - Loaded native gpl library 2010-12-07 15:49:44,705 INFO lzo.LzoCodec (LzoCodec.java:clinit(72)) - Successfully loaded initialized native-lzo library [hadoop-lzo rev c7acdaa96a7ce04538c0716fe699ffaf11836c70] 2010-12-07 15:49:44,712 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1 2010-12-07 15:49:44,880 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache. 2010-12-07 15:49:44,882 INFO exec.ExecDriver (SessionState.java:printInfo(268)) - Job running in-process (local Hadoop) 2010-12-07 15:49:44,882 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(256)) - job_local_0001 java.lang.RuntimeException: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:166) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) Caused by: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:157) ... 3 more First thing I noticed which is different from a successful job would be that it's trying to get a different jobid from the cache Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache I'm still confused. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967142#action_12967142 ] Bennie Schut commented on HIVE-1675: The good news is I'm unable to reproduce this once I set hive.exec.parallel=false. Perhaps it's cleaning uo some things during one of the steps while the other steps still need it? hive.exec.parallel works great for non local stuff for us but the combination of local and parallel seems like it has some issues. I was getting all kinds of different exceptions on local mode like the saxparser but also: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local_0001/attempt_local_0001_m_00_0/output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:50) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:193) and: java.io.FileNotFoundException: HIVE_PLAN1fe0a487-2e46-44e4-a557-9b3c9dddac07 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:155) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) I only get these issues when enabling parallel and local. At least an angle to look at. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
