[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548309#comment-13548309 ] Hudson commented on HIVE-3645: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-3645 : RCFileWriter does not implement the right function to support Federation (Arup Malakar via Ashutosh Chauhan) (Revision 1414286) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414286 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Labels: namenode_federation Fix For: 0.11.0 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534210#comment-13534210 ] Mikhail Bautin commented on HIVE-3645: -- I am getting the following compilation errors with this patch on branch-0.9: {code} ivy-retrieve-hadoop-shim: [echo] Project: shims [javac] Compiling 1 source file to /wd/hive/build/shims/classes [javac] /wd/hive/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:118: error: method getDefaultBlockSize in class FileSystem cannot be applied to given types; [javac] return fs.getDefaultBlockSize(path); [javac] ^ [javac] required: no arguments [javac] found: Path [javac] reason: actual and formal argument lists differ in length [javac] /wd/hive/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:123: error: method getDefaultReplication in class FileSystem cannot be applied to given types; [javac] return fs.getDefaultReplication(path); [javac] ^ [javac] required: no arguments [javac] found: Path [javac] reason: actual and formal argument lists differ in length [javac] 2 errors BUILD FAILED /wd/hive/build.xml:319: The following error occurred while executing this line: /wd/hive/build.xml:169: The following error occurred while executing this line: /wd/hive/shims/build.xml:90: The following error occurred while executing this line: /wd/hive/shims/build.xml:93: The following error occurred while executing this line: /wd/hive/shims/build.xml:82: Compile failed; see the compiler error output for details. {code} I am building with the default Hadoop version. RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534224#comment-13534224 ] Arup Malakar commented on HIVE-3645: From: https://issues.apache.org/jira/browse/HIVE-3754?focusedCommentId=13506596page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506596 You can use either of {code} ant clean package -Dhadoop.version=0.23.3 -Dhadoop-0.23.version=0.23.3 -Dhadoop.mr.rev=23 ant clean package -Dhadoop.version=2.0.0-alpha -Dhadoop-0.23.version=2.0.0-alpha -Dhadoop.mr.rev=23 {code} See HIVE-3754 for more details. I also see that default hadoop 23 version is 0.23.3 for branch-0.9 as well, so this should have worked without the arguments: {code} hadoop-0.23.version=0.23.3 {code} RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511006#comment-13511006 ] Hudson commented on HIVE-3645: -- Integrated in Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #219 (See [https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/219/]) HIVE-3645 : RCFileWriter does not implement the right function to support Federation (Arup Malakar via Ashutosh Chauhan) (Revision 1417220) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1417220 Files : * /hive/branches/branch-0.9/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/branches/branch-0.9/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/branches/branch-0.9/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/branches/branch-0.9/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/branches/branch-0.9/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/branches/branch-0.9/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511199#comment-13511199 ] Hudson commented on HIVE-3645: -- Integrated in Hive-0.9.1-SNAPSHOT-h0.21 #219 (See [https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/219/]) HIVE-3645 : RCFileWriter does not implement the right function to support Federation (Arup Malakar via Ashutosh Chauhan) (Revision 1417220) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1417220 Files : * /hive/branches/branch-0.9/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/branches/branch-0.9/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/branches/branch-0.9/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/branches/branch-0.9/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/branches/branch-0.9/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/branches/branch-0.9/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510061#comment-13510061 ] Ashutosh Chauhan commented on HIVE-3645: Committed to 0.9 branch. RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510143#comment-13510143 ] Arup Malakar commented on HIVE-3645: Thank you Ashutosh. RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505343#comment-13505343 ] Hudson commented on HIVE-3645: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3645 : RCFileWriter does not implement the right function to support Federation (Arup Malakar via Ashutosh Chauhan) (Revision 1414286) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414286 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504846#comment-13504846 ] Arup Malakar commented on HIVE-3645: Thanks Ashutosh for looking into the patch. If the branch patch looks fine can you please commit this to 0.9 branch as well? RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504419#comment-13504419 ] Ashutosh Chauhan commented on HIVE-3645: +1 will commit if tests pass. RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498262#comment-13498262 ] Arup Malakar commented on HIVE-3645: Looking at PIG-2791 looks like the following needs to be done: 1. Use getDefaultBlockSize(Path) and getDefaultReplication(Path) instead of getDefaultBlockSize() and getDefaultReplication(). As the ones without Path argument wont work in case of federated namenode. These methods need to be shimmed. 2. Bump hadoop dependency to 2.0.0-alpha as getDefaultBlockSize(Path)/getDefaultReplication(Path) are not available in 0.23.1 RCFileWriter does not implement the right function to support Federation Key: HIVE-3645 URL: https://issues.apache.org/jira/browse/HIVE-3645 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0, 0.10.0 Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10 Reporter: Viraj Bhat Create a table using Hive DDL {code} CREATE TABLE tmp_hcat_federated_numbers_part_1 ( id int, intnum int, floatnum float )partitioned by ( part1string, part2string ) STORED AS rcfile LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1'; {code} Populate it using Pig: {code} A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader(); B = filter A by id = 500; C = foreach B generate (int)id, (int)intnum, (float)floatnum; store C into 'default.tmp_hcat_federated_numbers_part_1' using org.apache.hcatalog.pig.HCatStorer ('part1=pig, part2=hcat_pig_insert', 'id: int,intnum: int,floatnum: float'); {code} Generates the following error when running on a Federated Cluster: {quote} 2012-10-29 20:40:25,011 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3 Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723) at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705) at org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86) at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100) at org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira