Sorry that was some bug at my writeFields method. its fixed now and I am able to load and dump the data. In SequenceFileLoader I have defined the corresponding keyconverter and value converter classes.
So, when I say raw = load 'in.txt' using SequenceFileLoader; dump raw It dumps the data but when I want to project the fields, it gives an error do i have to explicity specify the schema in load ? like: raw = load 'in.txt' using SequenceFileLoader as (t:(a:int, b:chararray,...)) On Wed, Oct 26, 2011 at 1:27 PM, Dmitriy Ryaboy <[email protected]> wrote: > What do you expect to see, how did you create it, and what are the weird > values? > Any chance your compression settings are different for writing and reading? > > On Tue, Oct 25, 2011 at 7:41 AM, Gayatri Rao <[email protected]> wrote: > > Thanks Dmitriy. > > I was trying to implement the MyClassConverter for my custom class > > and I overided and implemented the method > > > > @Override > > public Object bytesToObject(DataByteArray dataByteArray) throws > > IOException { > > > > MyClass o = (MyClass) ReflectionUtils.newInstance(MyClass.class, > > null); > > o.readFields(new DataInputStream(new > > ByteArrayInputStream(dataByteArray > > .get()))); > > return o; > > > > } > > > > and my MyClass.readFields is as follows: > > > > @Override > > public void readFields(DataInput in) throws IOException { > > num = in.readInt(); > > list = new ArrayList<String>(); > > for (int i = 0; i < 3; i++) { > > list.add(WritableUtils.readString(in)); > > } > > > > } > > > > This puts some weird data in num and list. Any idea what I might be doing > > wrong? > > > > > > On Tue, Oct 25, 2011 at 9:03 AM, Dmitriy Ryaboy <[email protected]> > wrote: > > > >> you can compile it with "ant -Dnothrift=true" > >> > >> There's also a "-Dnoprotobuf=true" option, but I just tried it and it > seems > >> we do require protobufs in 1 place that's not excluded when we skip > >> protocol > >> buffers, so you still need protoc version 2.3 > >> > >> D > >> > >> On Mon, Oct 24, 2011 at 6:52 PM, Gayatri Rao <[email protected]> > wrote: > >> > >> > Thats great, thanks, I ll check it out. Is thrift a dependency for > >> > building? > >> > > >> > On Mon, Oct 24, 2011 at 6:49 PM, Dmitriy Ryaboy <[email protected]> > >> > wrote: > >> > > >> > > Correct -- it's completely rewritten. > >> > > > >> > > We haven't published EB to a public maven repo, though I believe we > did > >> > add > >> > > a "maven-install" ant target to publish to your local maven repo. > >> > > > >> > > D > >> > > > >> > > On Mon, Oct 24, 2011 at 6:33 PM, Gayatri Rao <[email protected]> > >> > wrote: > >> > > > >> > > > I have checked the SequenceFileLoader from elephantbird and it > seems > >> > to > >> > > > use > >> > > > a different SequenceFileLoader as oppose to the one there is in > >> > piggybank > >> > > > Is there any reason for that? > >> > > > > >> > > > On Mon, Oct 24, 2011 at 5:57 PM, Gayatri Rao <[email protected] > > > >> > > wrote: > >> > > > > >> > > > > Thank Dmitriy. Are the jars available in maven repository? > >> > > > > > >> > > > > Thanks, > >> > > > > Gayatri > >> > > > > > >> > > > > > >> > > > > On Mon, Oct 24, 2011 at 11:55 AM, Dmitriy Ryaboy < > >> [email protected] > >> > > > >wrote: > >> > > > > > >> > > > >> We have a massively improved (well, rewritten from scratch) > >> > > > SequenceLoader > >> > > > >> in elephantbird. Take a look here: > >> > > > >> > >> > > > >> > >> > > > > >> > > > >> > > >> > https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/load/SequenceFileLoader.java > >> > > > >> > >> > > > >> No separate readme on usage, but all the related classes are > >> > > > >> well-documented > >> > > > >> in Javadocs. > >> > > > >> > >> > > > >> D > >> > > > >> > >> > > > >> On Mon, Oct 24, 2011 at 12:55 AM, Daniel Dai < > >> [email protected] > >> > > > >> > > > >> wrote: > >> > > > >> > >> > > > >> > I think it is the > >> > SequenceFileLoader.translateWritableToPigDataType > >> > > > >> which > >> > > > >> > does not support custom writable. Try to enhance > >> > > > >> > translateWritableToPigDataType. > >> > > > >> > > >> > > > >> > Daniel > >> > > > >> > > >> > > > >> > On Mon, Oct 24, 2011 at 12:21 AM, Gayatri Rao < > >> > [email protected]> > >> > > > >> wrote: > >> > > > >> > > >> > > > >> > > Hi All, > >> > > > >> > > > >> > > > >> > > I am trying to use the sequence file loader in piggybank > for > >> my > >> > > > custom > >> > > > >> > > writable object. I am working with pig 0.8, It looks like > it > >> > does > >> > > > not > >> > > > >> > work > >> > > > >> > > for user defined custom writables? > >> > > > >> > > Any pointers on how I can write a loader for my own custom > >> > > writable? > >> > > > >> > > > >> > > > >> > > Thanks, > >> > > > >> > > Gayatri > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
