Another thing to help me understand the Avro philosophy... When doing, public void put(int field$, java.lang.Object value$)
Why is field an integer? For instance, I have a String[] Column in my object. In protobuf, it would generate java methods like .putColumn(String[] item). Is there a reason avro can't do that? Or did I run the code generator in avro-tools wrong? If that doesn't work, could we generate an enum of field names to pass in, instead? I don't like having to know "Magic Numbers" :) Cheers, B On Tue, Jun 15, 2010 at 7:26 PM, Bradford Stephens <[email protected]> wrote: > That's.... erm, kinda bizarre. > > But hey, it works! Thanks! > > > > On Tue, Jun 15, 2010 at 6:56 PM, Scott Carey <[email protected]> wrote: >> Use GenericArray. The schema given to the generic array is not the schema >> of the member elements, but the actual array schema (yes it is confusing). >> >> new GenericData.Array<Utf8>(size, >> Schema.createArray(Schema.create(Type.STRING)); >> >> It would be useful to be able to simply use Utf8[] or List<Utf8> for the >> Specific API, but at this time it leverages GenericData. >> >> >> On Jun 15, 2010, at 6:25 PM, Bradford Stephens wrote: >> >>> That makes sense -- I'm getting errors during serialization, though. >>> >>> I convert my List<String> to Utf8[]. >>> >>> I then do a QueueItem.put() and get "Exception in thread "main" >>> java.lang.ClassCastException: [Lorg.apache.avro.util.Utf8; cannot be >>> cast to org.apache.avro.generic.GenericArray" >>> >>> How do I handle this Java->Avro interop? It seems to me that it should >>> be a lot simpler... >>> >>> If I try to create a GenericArray<Utf8> and add items to that, it >>> complains that my schema doesn't look right...so that doesn't feel >>> like the right path. >>> >>> My generated class looks like this: >>> >>> @SuppressWarnings("all") >>> public class QueueItem extends >>> org.apache.avro.specific.SpecificRecordBase implements >>> org.apache.avro.specific.SpecificRecord { >>> public static final org.apache.avro.Schema SCHEMA$ = >>> org.apache.avro.Schema.parse("{\"type\":\"record\",\"name\":\"QueueItem\",\"namespace\":\"com.dts\",\"fields\":[{\"name\":\"Columns\",\"type\":[\"null\",{\"type\":\"array\",\"items\":\"string\"}]}]}"); >>> >>> public org.apache.avro.generic.GenericArray<org.apache.avro.util.Utf8> >>> Columns; >>> public org.apache.avro.Schema getSchema() { return SCHEMA$; } >>> public java.lang.Object get(int field$) { >>> >>> >>> switch (field$) { >>> case 0: return Columns; >>> default: throw new org.apache.avro.AvroRuntimeException("Bad index"); >>> } >>> } >>> �...@suppresswarnings(value="unchecked") >>> public void put(int field$, java.lang.Object value$) { >>> switch (field$) { >>> case 0: Columns = >>> (org.apache.avro.generic.GenericArray<org.apache.avro.util.Utf8>)value$; >>> break; >>> default: throw new org.apache.avro.AvroRuntimeException("Bad index"); >>> } >>> } >>> } >>> >>> >>> >>> >>> On Tue, Jun 15, 2010 at 8:57 AM, Philip Zeyliger <[email protected]> >>> wrote: >>>> Hi Bradford, >>>> I believe you use a SpecificDatumReader. Something like: >>>> >>>> final static SpecicificDatumReader<QueueItem> QUEUE_ITEM_READER = new >>>> SpecificDatumReader<QueueItem>(QueueItem.class); >>>> QueueItem q = QUEUE_ITEM_READER.read(null, decoder); >>>> There doesn't seem to be a test that exercises this code path in an >>>> explanatory way, but java/src/java/org/apache/avro/ipc/Requestor.java uses >>>> something quite similar. >>>> -- Philip >>>> >>>> On Mon, Jun 14, 2010 at 9:20 PM, Bradford Stephens >>>> <[email protected]> wrote: >>>>> >>>>> Greetings, >>>>> >>>>> I've poked around for examples of this, but I can't find any. I >>>>> imagine it's a fairly common use case. >>>>> >>>>> I'm serializing some simple objects into bytes for placement onto >>>>> RabbitMQ. My java class is pretty simple (but it'll grow in complexity >>>>> in time).: >>>>> >>>>> { >>>>> String[] Columns; >>>>> } >>>>> >>>>> >>>>> So, I made a .json schema containing this: >>>>> { >>>>> "namespace": "com.dts", >>>>> "name": "QueueItem", >>>>> "type": "record", >>>>> "fields": [ >>>>> {"name": "Columns", "type": ["null", {"type": "array", >>>>> "items":"string"}]} >>>>> ] >>>>> } >>>>> >>>>> >>>>> And generated a java class ... >>>>> >>>>> Now, I'm writing a test to serialize and deserialize some items. I can >>>>> figure out the serialization, but not deserialization back to objects. >>>>> >>>>> Schema s = Schema.parse(new File("queuetype.json")); >>>>> >>>>> ByteArrayOutputStream bao = new ByteArrayOutputStream(); >>>>> GenericDatumWriter w = new GenericDatumWriter(s); >>>>> Encoder e = new BinaryEncoder(bao); >>>>> e.init (bao); >>>>> >>>>> >>>>> QueueItem r = new QueueItem(); >>>>> r.put(0, items); >>>>> w.write(r, e); >>>>> e.flush(); >>>>> >>>>> ByteArrayInputStream is = new >>>>> ByteArrayInputStream(bao.toByteArray()); >>>>> DecoderFactory df = new DecoderFactory(); >>>>> Decoder d = df.createBinaryDecoder(is, null); >>>>> >>>>> QueueItem itemout = (QueueItem) ..... >>>>> >>>>> >>>>> And that's what I can't figure out -- how do I use a decoder method to >>>>> create QueueItems? >>>>> >>>>> Cheers, >>>>> B >>>>> >>>>> radford Stephens, >>>>> Founder, Drawn to Scale >>>>> drawntoscalehq.com >>>>> 727.697.7528 >>>>> >>>>> http://www.drawntoscalehq.com -- The intuitive, cloud-scale data >>>>> solution. Process, store, query, search, and serve all your data. >>>>> >>>>> http://www.roadtofailure.com -- The Fringes of Scalability, Social >>>>> Media, and Computer Science >>>> >>>> >>> >>> >>> >>> -- >>> Bradford Stephens, >>> Founder, Drawn to Scale >>> drawntoscalehq.com >>> 727.697.7528 >>> >>> http://www.drawntoscalehq.com -- The intuitive, cloud-scale data >>> solution. Process, store, query, search, and serve all your data. >>> >>> http://www.roadtofailure.com -- The Fringes of Scalability, Social >>> Media, and Computer Science >> >> > > > > -- > Bradford Stephens, > Founder, Drawn to Scale > drawntoscalehq.com > 727.697.7528 > > http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > solution. Process, store, query, search, and serve all your data. > > http://www.roadtofailure.com -- The Fringes of Scalability, Social > Media, and Computer Science > -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com -- The intuitive, cloud-scale data solution. Process, store, query, search, and serve all your data. http://www.roadtofailure.com -- The Fringes of Scalability, Social Media, and Computer Science
