Thanks. :) That helped a lot.
-Rishav
On Tue, Apr 20, 2010 at 10:42 PM, J.G.Konrad <[email protected]> wrote:
> The LexicalizedParser does not know how to read from the dfs. You could
> leverage the distributed cache to copy the grammar to the work nodes and
> load them that way. That would look something like
>
> // initialize cache with file from dfs
> DistributedCache.addCacheFile(new
> URI("/user/me/project4/englishPCFG.ser.gz"), conf);
>
> // in Map configure
> Path[] ps = DistributedCache.getLocalCacheFiles(conf);
> // assuming only one file
> lp = new LexicalizedParser(ps[0].toString());
>
> Another option would be to server your grammar from a webserver so then
> you could just give you lp a url and it will download and load the grammer.
> I don't know if you have access to a webserver to do that. If you do have
> access then you would load the grammer like
>
> lp = new LexicalizedParser("http://some-domain/englishPCFG.ser.gz";)
>
>
>
> On Tue, Apr 20, 2010 at 12:18 PM, Rishav <[email protected]> wrote:
>
>> So My mapper has:
>> public static LexicalizedParser lp;
>>
>> public void configure(JobConf job) {
>> lp = new
>> LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
>> }
>>
>>
>> I get a runtimeException of error in configuration object. If you notice
>> later on in the exception thread, the parser actually gets loaded. My hunch
>> is it is happening in the master node which can access it. The
>> englishPCFG.ser.gz trained model is on hdfs.
>>
>>
>>
>> 10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
>> attempt_201003091315_0078_m_000000_0, Status : FAILED
>> java.lang.RuntimeException: Error in configuring object
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 5 more
>> Caused by: java.lang.RuntimeException: Error in configuring object
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> ... 10 more
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 13 more
>> Caused by: java.lang.RuntimeException: File not found:
>> /user/me/project4/englishPCFG.ser.gz
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
>> at ParseMapper.configure(ParseMapper.java:22)
>> ... 18 more
>> Caused by: java.io.FileNotFoundException:
>> /user/me/project4/englishPCFG.ser.gz (No such file or directory)
>> at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:106)
>> at java.io.FileInputStream.<init>(FileInputStream.java:66)
>> at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
>> ... 22 more
>>
>> attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
>> /user/me/project4/englishPCFG.ser.gz ...
>> 10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
>> attempt_201003091315_0078_m_000001_0, Status : FAILED
>> java.lang.RuntimeException: Error in configuring object
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 5 more
>> Caused by: java.lang.RuntimeException: Error in configuring object
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> ... 10 more
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 13 more
>> Caused by: java.lang.RuntimeException: File not found:
>> /user/me/project4/englishPCFG.ser.gz
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
>> at ParseMapper.configure(ParseMapper.java:22)
>> ... 18 more
>> Caused by: java.io.FileNotFoundException:
>> /user/me/project4/englishPCFG.ser.gz (No such file or directory)
>> at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:106)
>> at java.io.FileInputStream.<init>(FileInputStream.java:66)
>> at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
>> at
>> edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
>> ... 22 more
>>
>> attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
>> /user/me/project4/englishPCFG.ser.gz ...
>> 10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
>> attempt_201003091315_0078_m_000000_1, Status : FAILED
>> java.lang.RuntimeException: Error in configuring object
>>
>>
>> On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad <[email protected]> wrote:
>>
>>> I would suggest adding a configure() method to your Map class and then
>>> load the parser and save it in a class object.
>>>
>>>
>>> On Tue, Apr 20, 2010 at 4:16 AM, Rishav <[email protected]> wrote:
>>>
>>>> Hello
>>>>
>>>> I am trying to pass an object of type LexicalizedParser (which is from
>>>> an imported jar file stanford-parser) from the main method to the Mapper
>>>> Class. This is to load the trained model first and then apply on every file
>>>> offset in the map function in the Mapper Class.
>>>> Could you tell me the best way to pass an object of any type to the
>>>> Mapper?
>>>>
>>>> Thanking you
>>>> Rishav
>>>>
>>>>
>>>> --
>>>> Rishav Bhowmick
>>>> Computer Science -2010
>>>> Carnegie Mellon University
>>>> Tel: +974-561-4371
>>>> e-mail: [email protected]
>>>>
>>>>
>>>
>>
>>
>> --
>> Rishav Bhowmick
>> Computer Science -2010
>> Carnegie Mellon University
>> Tel: +974-561-4371
>> e-mail: [email protected]
>>
>>
>
--
Rishav Bhowmick
Computer Science -2010
Carnegie Mellon University
Tel: +974-561-4371
e-mail: [email protected]
