Thanks for the response. Still no solution, here is where I am at...
- I wasn't explicitly specifying a schema when I got this error... - I tried changing the generate statements, 1 field ones succeed, but others failed to allow Json storage due to the same parse schema error. - next: How can I know If a schema is valid for json storage or not? From the docs, I think any combo of primitives, maps, bags, and tuples should be ok JsonStorage? After all : each maps to a well defined json structure. How would i determine if this is a bug in pig? If it is I will file a nice yummy jira :) On Dec 15, 2013, at 1:24 AM, Cheolsoo Park <[email protected]> wrote: >>> Syntax error, unexpected symbol at or near ',' > > From the stack trace, the QueryParser got an invalid token for the schema > string, and I believe JsonStorage got into this error because it implements > checkSchema()<https://github.com/apache/pig/blob/trunk/src/org/apache/pig/builtin/JsonStorage.java#L108>method > unlike PigStorage. PigStorage stores everything in plain text, so it > doesn't need to know any schema. But JsonStorage does. > > Now it seems that your invalid schema string is from this > line<https://github.com/apache/pig/blob/trunk/src/org/apache/pig/newplan/logical/rules/InputOutputFileValidator.java#L65>. > So my guess is that somehow your store operator is given an invalid schema > for "uniqcnt". If you set a breakpoint at this line in IDE, you should be > able to see what it's like, and you can go from there. It could be a bug in > Pig. :-) > > >> On Thu, Dec 12, 2013 at 8:58 PM, Jay Vyas <[email protected]> wrote: >> >> Hi pig ! >> >> Should JsonStorage support every type of store() that PigStorage supports? >> It appears that it breaks in a clear case where PigStorage works... here >> are the details. >> >> I'm able to serialize one of my datasets using PigStorage, but not >> JsonStorage. >> >> <<<< THIS WORKS >>>> >> >> *1) pigServer.store("uniqcnt", "/tmp/bbb1", "PigStorage");* >> >> results in this file: >> >>> cat /tmp/bbb >> >> storeCode_AK 4 >> storeCode_AZ 2 >> storeCode_CO 7 >> storeCode_CT 4 >> storeCode_OK 2 >> >> <<<< THIS FAILS >>>> >> >> *2) pigServer.store("uniqcnt", "/tmp/bbb2", "JsonStorage");* >> >> Results in a stacktrace : >> >> When I try with JSonStorage: >> >> Failed to parse: <line 1, column 6> Syntax error, unexpected symbol at or >> near ',' >> at >> >> org.apache.pig.parser.QueryParserDriver.parseSchema(QueryParserDriver.java:94) >> at >> >> org.apache.pig.parser.QueryParserDriver.parseSchema(QueryParserDriver.java:108) >> at org.apache.pig.impl.util.Utils.parseSchema(Utils.java:208) >> at org.apache.pig.impl.util.Utils.getSchemaFromString(Utils.java:182) >> at >> org.apache.pig.builtin.JsonStorage.prepareToWrite(JsonStorage.java:140) >> at >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:125) >> at >> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86) >> at >> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:553) >> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216) >>
