The LexicalizedParser does not know how to read from the dfs. You could
leverage the distributed cache to copy the grammar to the work nodes and
load them that way. That would look something like
// initialize cache with file from dfs
DistributedCache.addCacheFile(new
URI("/user/me/project4/englishPCFG.ser.gz"), conf);
// in Map configure
Path[] ps = DistributedCache.getLocalCacheFiles(conf);
// assuming only one file
lp = new LexicalizedParser(ps[0].toString());
Another option would be to server your grammar from a webserver so then you
could just give you lp a url and it will download and load the grammer. I
don't know if you have access to a webserver to do that. If you do have
access then you would load the grammer like
lp = new LexicalizedParser("http://some-domain/englishPCFG.ser.gz";)
On Tue, Apr 20, 2010 at 12:18 PM, Rishav <[email protected]> wrote:
> So My mapper has:
> public static LexicalizedParser lp;
>
> public void configure(JobConf job) {
> lp = new LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
> }
>
>
> I get a runtimeException of error in configuration object. If you notice
> later on in the exception thread, the parser actually gets loaded. My hunch
> is it is happening in the master node which can access it. The
> englishPCFG.ser.gz trained model is on hdfs.
>
>
>
> 10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
> attempt_201003091315_0078_m_000000_0, Status : FAILED
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 5 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 10 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 13 more
> Caused by: java.lang.RuntimeException: File not found:
> /user/me/project4/englishPCFG.ser.gz
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
> at ParseMapper.configure(ParseMapper.java:22)
> ... 18 more
> Caused by: java.io.FileNotFoundException:
> /user/me/project4/englishPCFG.ser.gz (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:106)
> at java.io.FileInputStream.<init>(FileInputStream.java:66)
> at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
> ... 22 more
>
> attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
> /user/me/project4/englishPCFG.ser.gz ...
> 10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
> attempt_201003091315_0078_m_000001_0, Status : FAILED
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 5 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 10 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 13 more
> Caused by: java.lang.RuntimeException: File not found:
> /user/me/project4/englishPCFG.ser.gz
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
> at ParseMapper.configure(ParseMapper.java:22)
> ... 18 more
> Caused by: java.io.FileNotFoundException:
> /user/me/project4/englishPCFG.ser.gz (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:106)
> at java.io.FileInputStream.<init>(FileInputStream.java:66)
> at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
> at
> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
> ... 22 more
>
> attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
> /user/me/project4/englishPCFG.ser.gz ...
> 10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
> attempt_201003091315_0078_m_000000_1, Status : FAILED
> java.lang.RuntimeException: Error in configuring object
>
>
> On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad <[email protected]> wrote:
>
>> I would suggest adding a configure() method to your Map class and then
>> load the parser and save it in a class object.
>>
>>
>> On Tue, Apr 20, 2010 at 4:16 AM, Rishav <[email protected]> wrote:
>>
>>> Hello
>>>
>>> I am trying to pass an object of type LexicalizedParser (which is from an
>>> imported jar file stanford-parser) from the main method to the Mapper Class.
>>> This is to load the trained model first and then apply on every file offset
>>> in the map function in the Mapper Class.
>>> Could you tell me the best way to pass an object of any type to the
>>> Mapper?
>>>
>>> Thanking you
>>> Rishav
>>>
>>>
>>> --
>>> Rishav Bhowmick
>>> Computer Science -2010
>>> Carnegie Mellon University
>>> Tel: +974-561-4371
>>> e-mail: [email protected]
>>>
>>>
>>
>
>
> --
> Rishav Bhowmick
> Computer Science -2010
> Carnegie Mellon University
> Tel: +974-561-4371
> e-mail: [email protected]
>
>
