Re: Schema exclusion from Avro message

I'm planning to stream it via Kafka. I think I've sorted it out. The Object Container Files is not an essential part of the Avro. Instead of using DataFileReader/DataFileWriter, I can use DatumReader/DatumWriter to encode and decode the binary data and store the schema separately. Wai Yip

Re: Clarification of getSchema()#getIndexNamed(fieldName) in Avro 1.7.5 Java API

Hey Robert, > > But this is not what you want, I think, so the following is the best way > as you have discovered: > > int fieldIndex = persistent.getSchema().getField(fieldName).pos(); > > > Lewis > >> > > > -- > Yeah thanks for the explanation. It's working like a charm now. Thanks very much

Re: Clarification of getSchema()#getIndexNamed(fieldName) in Avro 1.7.5 Java API

Hi Doug, On Mon, Jan 27, 2014 at 8:05 PM, Doug Cutting wrote: > > Does this help? > > > In short... yes :) it does help. I know understand that I was using #getIndexNamed("string"); incorrectly... or rather correctly but not for what I was wanting to do! Thanks for feedback. Lewis -- *Lewis*

Re: Clarification of getSchema()#getIndexNamed(fieldName) in Avro 1.7.5 Java API

Hi Lewis, The method *getIndexNamed* is only used for *Union* schemas and gives the index of the union's type, so 0 for "null" and 1 for "string" in this case. The *persistent.getSchema()* returns the *Record Schema* in this case and the get *getField* method on that gets the field. *Fields* howev

Re: Clarification of getSchema()#getIndexNamed(fieldName) in Avro 1.7.5 Java API

It's not clear to me what you're trying to do. "Persistent" is a record with a field named "url" whose value is a union. Schema recordSchema = Persistent.class.getSchema(); List fields = recordSchema.getFields(); Field urlField = recordSchema.getField("url"); // a field's position is within the

Re: Schema exclusion from Avro message

If you're using Avro's RPC mechanism, schemas are only sent when the client and server do not already have each other's schema. Each client request is preceded by a hash of the clients schema and the schema it thinks the server is using. If the server already has the client's schema, and the clie

Clarification of getSchema()#getIndexNamed(fieldName) in Avro 1.7.5 Java API

Hi Folks, I have the following UNION field {"name": "url", "type": ["null","string"], "default":null}, Once I've generated the Persistent class, I set a vlue for this field as follows page.setUrl(new Utf8("example.org")); When however I attempt to get the field position of the URL UNION with i

Re: [ANNOUNCE] Avro release 1.7.6

On Mon, Jan 27, 2014 at 11:27 AM, Sid Shetye wrote: > In a similar vein, how about publishing the C# binaries to NuGet? That would be great to add. Currently it doesn't appear the build generates a .nupkg file. If someone supplies a patch so that these are created then we can upload them to NuG

Re: [ANNOUNCE] Avro release 1.7.6

Thanks Doug, Yes, the name is fine! It looks like I did something wrong with the setup.py… still learning! I'll send a fix for that shortly. C. On Mon, Jan 27, 2014 at 11:25 AM, Doug Cutting wrote: > On Sat, Jan 25, 2014 at 8:58 PM, Christophe Taton > wrote: > > Is it also possible to push the

RE: [ANNOUNCE] Avro release 1.7.6

Hey Doug, In a similar vein, how about publishing the C# binaries to NuGet? Cheers Sid -Original Message- From: Doug Cutting [mailto:cutt...@apache.org] Sent: Monday, January 27, 2014 11:25 AM To: user@avro.apache.org Subject: Re: [ANNOUNCE] Avro release 1.7.6 On Sat, Jan 25, 2014 at 8

Re: [ANNOUNCE] Avro release 1.7.6

On Sat, Jan 25, 2014 at 8:58 PM, Christophe Taton wrote: > Is it also possible to push the Python3 version to PyPi? I just did that. https://pypi.python.org/pypi/avro-python3/1.7.6 I renamed it avro-python3. Does that seem reasonable? It seems like one of several conventions used. Doug

Schema exclusion from Avro message

I found Deepesh's question back in December. I have joined the mailing list later. So don't have the message in my inbox and I do not know the proper way to reply. Anyway I have include the original message below. I have the similar issue. In addition I'm interested to find out about Python an

Re: Enum & backward compatibility in distributed services...

You'd like the compile-time type-checking of specific, but the run-time flexibility of generic, right? Here's a way we might achieve this. Given the following schemas: {"type":"enum", "name":"Color", "symbols":["RED", "GREEN", "BLUE"]} {"type":"record", "name":"Shape", "fields":[ {"name":"xPo

Re: data missing in writing an AVRO file.

Hi, This is most likely related to this issue https://issues.apache.org/jira/browse/AVRO-1364. It is fixed in Avro 1.7.6, so first try updating your Avro-C lib. -Mika On Jan 27, 2014, at 6:35 PM, Amrith Kumar wrote: > Here is some additional debugging information … > > I created this simpl

RE: data missing in writing an AVRO file.

Here is some additional debugging information ... I created this simple CSV file that looks thus. ubuntu@petest1:/mnt/avrotest$ head maketest.csv "data1", "data2", 0, 1804289383, 1, 846930886, 2, 1681692777, 3, 1714636915, 4, 1957747793, 5, 424238335, 6, 719885386, 7, 1649760492, 8, 596516649, ub

data missing in writing an AVRO file.

Greetings, I'm attempting to convert some very large CSV files into AVRO format. To this end, I wrote a csvtoavro converter using C API v1.7.5. The essence of the program is this: // initialize line counter lineno = 0; // make a schema first avro_schema_from_json_length (...); // make a gener