Would it be fair to assume that SchemaUtil.newBagSchema(...) etc. will do the right thing for whichever version you're using, without requiring additional manual intervention?
On 31 October 2011 23:20, Dmitriy Ryaboy <[email protected]> wrote: > Our 0.8-compatible schemas for nested structures stopped working on > 0.9 (it wasn't just two-level access, it's also how deeply nested > structures are treated). > You can see what the problem is from EB's code that uses this method: > > https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java > > (read singleFieldToTupleSchema method) > > D > > > On Mon, Oct 31, 2011 at 3:05 PM, Daniel Dai <[email protected]> wrote: >> I guess this is what Dmitriy mean: since it is marked as deprecated, Pig >> could drop it anytime in the future. So you'd better make your UDF work >> even if twoLevelAccess is dropped. >> >> Daniel >> >> On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <[email protected] >>> wrote: >> >>> Thanks chaps. Just to clarify something though -- Daniel's answer >>> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's >>> advice suggests that it should actively be avoided. Unless I'm >>> misreading what one of you means? >>> >>> On 31 October 2011 20:34, Dmitriy Ryaboy <[email protected]> wrote: >>> > When implementing Daniel's suggestion, future-proof yourself: >>> > >>> > >>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java >>> > >>> > (see the Pig9orNewer() method) >>> > >>> > D >>> > >>> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <[email protected]> >>> wrote: >>> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it >>> >> any more. >>> >> >>> >> Daniel >>> >> >>> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg < >>> >> [email protected]> wrote: >>> >> >>> >>> Hi, >>> >>> >>> >>> If I create two schemas like this: >>> >>> >>> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}"); >>> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP}); >>> >>> >>> >>> then compare them like this: >>> >>> >>> >>> boolean match = Schema.equals(expected, output, false, true); >>> >>> >>> >>> they don't match, even though the toString output of the two is >>> identical: >>> >>> >>> >>> {b: {t: (f0: map[ ])}} >>> >>> >>> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is >>> >>> true, but in s2, it's false. >>> >>> >>> >>> Which is correct? If I'm writing a UDF that returns a bag of maps, >>> >>> what should I make the outputSchema method return? >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Andrew. >>> >>> >>> >>> -- >>> >>> >>> >>> http://tinyurl.com/andrew-clegg-linkedin | >>> http://twitter.com/andrew_clegg >>> >>> >>> >> >>> > >>> >>> >>> >>> -- >>> >>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg >>> >> > -- http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
