Our 0.8-compatible schemas for nested structures stopped working on 0.9 (it wasn't just two-level access, it's also how deeply nested structures are treated). You can see what the problem is from EB's code that uses this method:
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java (read singleFieldToTupleSchema method) D On Mon, Oct 31, 2011 at 3:05 PM, Daniel Dai <[email protected]> wrote: > I guess this is what Dmitriy mean: since it is marked as deprecated, Pig > could drop it anytime in the future. So you'd better make your UDF work > even if twoLevelAccess is dropped. > > Daniel > > On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <[email protected] >> wrote: > >> Thanks chaps. Just to clarify something though -- Daniel's answer >> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's >> advice suggests that it should actively be avoided. Unless I'm >> misreading what one of you means? >> >> On 31 October 2011 20:34, Dmitriy Ryaboy <[email protected]> wrote: >> > When implementing Daniel's suggestion, future-proof yourself: >> > >> > >> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java >> > >> > (see the Pig9orNewer() method) >> > >> > D >> > >> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <[email protected]> >> wrote: >> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it >> >> any more. >> >> >> >> Daniel >> >> >> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg < >> >> [email protected]> wrote: >> >> >> >>> Hi, >> >>> >> >>> If I create two schemas like this: >> >>> >> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}"); >> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP}); >> >>> >> >>> then compare them like this: >> >>> >> >>> boolean match = Schema.equals(expected, output, false, true); >> >>> >> >>> they don't match, even though the toString output of the two is >> identical: >> >>> >> >>> {b: {t: (f0: map[ ])}} >> >>> >> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is >> >>> true, but in s2, it's false. >> >>> >> >>> Which is correct? If I'm writing a UDF that returns a bag of maps, >> >>> what should I make the outputSchema method return? >> >>> >> >>> Thanks, >> >>> >> >>> Andrew. >> >>> >> >>> -- >> >>> >> >>> http://tinyurl.com/andrew-clegg-linkedin | >> http://twitter.com/andrew_clegg >> >>> >> >> >> > >> >> >> >> -- >> >> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg >> >
