Hi, I want to know , when should i use Mapper , Reduce and Combiner.
What all methods are there in them. Please suggest for study in detail. As I am fresher . Thanks in advance Ranjini On Tue, Dec 17, 2013 at 2:34 PM, unmesha sreeveni <[email protected]>wrote: > Ranjini can u pls check this. This is not perfect ..I simply did to > check my xml data. > > https://github.com/studhadoop/xmlparsing-hadoop/blob/master/XmlParser11.java > > > On Tue, Dec 17, 2013 at 2:26 PM, Ranjini Rathinam > <[email protected]>wrote: > >> Hi, >> >> The driver class and my Mapper class i have used >> org.apache.hadoop.mapreduce.lib >> >> and in the XmlInputFormat.java class also i have used the >> org.apache.hadoop.mapreduce.lib >> >> but still iam getting this error. >> >> Please suggest. >> >> Thanks in advance >> >> Ranjini >> >> On Tue, Dec 17, 2013 at 2:07 PM, Shekhar Sharma >> <[email protected]>wrote: >> >>> Hello Ranjini, >>> This error will come when you use mix and match newer and older API.. >>> >>> You might have written program using newer API and the the XML input >>> format is using older api.. >>> The older api has package structure of org.apache.hadoop.mapred >>> >>> The newer api has package structure package of >>> org.apache.hadoop.mapreduce.lib >>> >>> Check out the XMLINputFormat.java, which package of FileInputFormat >>> they have used... >>> >>> >>> Regards, >>> Som Shekhar Sharma >>> +91-8197243810 >>> >>> >>> On Tue, Dec 17, 2013 at 12:55 PM, Ranjini Rathinam >>> <[email protected]> wrote: >>> > Hi, >>> > >>> > I am using hadoop 0.20 version >>> > >>> > In that while exceuting the XmlInformat class >>> > I am getting the error as >>> > >>> > "Error: Found Class org.apache.hadoop.mapreduce.TaskAttemptContext, >>> but >>> > interface was excepted,." >>> > >>> > Please suggest to fix the error. >>> > >>> > Thanks in advance. >>> > >>> > Ranjini >>> > >>> > On Wed, Dec 11, 2013 at 12:30 PM, Ranjini Rathinam < >>> [email protected]> >>> > wrote: >>> >> >>> >> hi, >>> >> >>> >> I have fixed the error , the code is running fine, but this code just >>> >> split the part of the tag. >>> >> >>> >> i want to convert into text format so that i can load them into >>> tables of >>> >> hbase and hive. >>> >> >>> >> I have used the DOM Parser but this parser uses File as Object but >>> hdfs >>> >> uses FileSystem. >>> >> >>> >> Eg, >>> >> >>> >> File fXmlFile = new File("D:/elango/test.xml"); >>> >> >>> >> System.out.println(g); >>> >> DocumentBuilderFactory dbFactory = >>> DocumentBuilderFactory.newInstance(); >>> >> DocumentBuilder dBuilder = dbFactory.newDocumentBuilder(); >>> >> Document doc = dBuilder.parse(fXmlFile); >>> >> >>> >> >>> >> This cant be used as hdfs, because hdfs path is accessed through >>> >> FileSystem. >>> >> >>> >> I kindly request u to , Please suggest me to fix the above issue. >>> >> >>> >> Thanks in advance >>> >> >>> >> Ranjini R >>> >> >>> >> >>> >> >>> >> >>> >> On Tue, Dec 10, 2013 at 11:07 AM, Ranjini Rathinam >>> >> <[email protected]> wrote: >>> >>> >>> >>> >>> >>> >>> >>> ---------- Forwarded message ---------- >>> >>> From: Shekhar Sharma <[email protected]> >>> >>> Date: Mon, Dec 9, 2013 at 10:23 PM >>> >>> Subject: Re: Hadoop-MapReduce >>> >>> To: [email protected] >>> >>> Cc: [email protected] >>> >>> >>> >>> >>> >>> It does work i have used it long back.. >>> >>> >>> >>> BTW if it is not working, write the custom input format and implement >>> >>> your record reader. That would be far more easy than breaking your >>> >>> head with others code. >>> >>> >>> >>> Break your problem in step: >>> >>> >>> >>> (1) First the XML data is multiline...Meaning multiple lines makes a >>> >>> single record for you...May be a record for you would be >>> >>> >>> >>> <person> >>> >>> <fname>x</fname> >>> >>> <lname>y</lname> >>> >>> </person> >>> >>> >>> >>> (2) Implement a record reader that looks out for the starting and >>> >>> ending person tag ( Checkout how RecordReader.java is written) >>> >>> >>> >>> (3) Once you got the contents between starting and ending tag, now >>> you >>> >>> can use a xml parser to parse the contents into an java object and >>> >>> form your own key value pairs ( custom key and custom value) >>> >>> >>> >>> >>> >>> Hope you have enough pointers to write the code. >>> >>> >>> >>> >>> >>> Regards, >>> >>> Som Shekhar Sharma >>> >>> +91-8197243810 >>> >>> >>> >>> >>> >>> On Mon, Dec 9, 2013 at 6:30 PM, Ranjini Rathinam < >>> [email protected]> >>> >>> wrote: >>> >>> > Hi Subroto Sanyal, >>> >>> > >>> >>> > The link provided about xml, it does not work . The Class written >>> >>> > XmlContent is not allowed in the XmlInputFormat. >>> >>> > >>> >>> > I request you to help , whether this scenaio some one has coded, >>> and >>> >>> > needed >>> >>> > working code. >>> >>> > >>> >>> > I have written using SAX Parser too, but eventhough the jars are >>> added >>> >>> > in >>> >>> > classpath THe error is is coming has NoClasFoung Exception. >>> >>> > >>> >>> > Please provide sample code for the same. >>> >>> > >>> >>> > Thanks in advance, >>> >>> > Ranjini.R >>> >>> > >>> >>> > On Mon, Dec 9, 2013 at 12:34 PM, Ranjini Rathinam >>> >>> > <[email protected]> >>> >>> > wrote: >>> >>> >> >>> >>> >> >>> >>> >>>> Hi, >>> >>> >>>> >>> >>> >>>> As suggest by the link below , i have used for my program , >>> >>> >>>> >>> >>> >>>> but i am facing the below issues, please help me to fix these >>> error. >>> >>> >>>> >>> >>> >>>> >>> >>> >>>> XmlReader.java:8: XmlReader.Map is not abstract and does not >>> >>> >>>> override >>> >>> >>>> abstract method >>> >>> >>>> >>> >>> >>>> >>> map(org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>,org.apache.hadoop.mapred.Reporter) >>> >>> >>>> in org.apache.hadoop.mapred.Mapper >>> >>> >>>> public static class Map extends MapReduceBase implements Mapper >>> >>> >>>> <LongWritable, Text, Text, Text> { >>> >>> >>>> ^ >>> >>> >>>> ./XmlInputFormat.java:16: XmlInputFormat.XmlRecordReader is not >>> >>> >>>> abstract >>> >>> >>>> and does not override abstract method >>> >>> >>>> next(java.lang.Object,java.lang.Object) in >>> >>> >>>> org.apache.hadoop.mapred.RecordReader >>> >>> >>>> public class XmlRecordReader implements RecordReader { >>> >>> >>>> ^ >>> >>> >>>> Note: XmlReader.java uses unchecked or unsafe operations. >>> >>> >>>> Note: Recompile with -Xlint:unchecked for details. >>> >>> >>>> 2 errors >>> >>> >>>> >>> >>> >>>> >>> >>> >>>> i am using hadoop 0.20 version and java 1.6 . >>> >>> >>>> >>> >>> >>>> Please suggest. >>> >>> >>>> >>> >>> >>>> Thanks in advance. >>> >>> >>>> >>> >>> >>>> Regrads, >>> >>> >>>> Ranjini. R >>> >>> >>>> On Mon, Dec 9, 2013 at 11:08 AM, Ranjini Rathinam >>> >>> >>>> <[email protected]> wrote: >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> ---------- Forwarded message ---------- >>> >>> >>>>> From: Subroto <[email protected]> >>> >>> >>>>> Date: Fri, Dec 6, 2013 at 4:42 PM >>> >>> >>>>> Subject: Re: Hadoop-MapReduce >>> >>> >>>>> To: [email protected] >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> Hi Ranjini, >>> >>> >>>>> >>> >>> >>>>> A good example to look into : >>> >>> >>>>> http://www.undercloud.org/?p=408 >>> >>> >>>>> >>> >>> >>>>> Cheers, >>> >>> >>>>> Subroto Sanyal >>> >>> >>>>> >>> >>> >>>>> On Dec 6, 2013, at 12:02 PM, Ranjini Rathinam wrote: >>> >>> >>>>> >>> >>> >>>>> Hi, >>> >>> >>>>> >>> >>> >>>>> How to read xml file via mapreduce and load them in hbase and >>> hive >>> >>> >>>>> using java. >>> >>> >>>>> >>> >>> >>>>> Please provide sample code. >>> >>> >>>>> >>> >>> >>>>> I am using hadoop 0.20 version and java 1.6. Which parser >>> version >>> >>> >>>>> should be used. >>> >>> >>>>> >>> >>> >>>>> Thanks in advance. >>> >>> >>>>> >>> >>> >>>>> Ranjini >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>> >>> >>> >>> >>> >>> >> >>> >>> > >>> >>> >>> >> >>> > >>> >> >> > > > -- > *Thanks & Regards* > > Unmesha Sreeveni U.B > > *Junior Developer* > > >
