Thanks Scott, I figured it out. The problem was simply an improper loop
design.
I corrected the loop code and now I get an array of Ifnet records.
[amiller@loclhost $] avro cat --fields hostname,ifnet data_2013-01-08.avro
{"hostname": "host08", "ifnet": [{"send_bps": 0, "recv_bps": 0, "name":
"e0P"}, {"send_bps": 171256, "recv_bps": 197512, "name": "vif1"},
{"send_bps": 0, "recv_bps": 0, "name": "vif2"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}]}
{"hostname": "host11", "ifnet": [{"send_bps": 9992353544, "recv_bps":
14049010776, "name": "e1a"}, {"send_bps": 0, "recv_bps": 0, "name": "e1b"},
{"send_bps": 2736, "recv_bps": 2960, "name": "e0P"}, {"send_bps": 9376,
"recv_bps": 19312, "name": "c0a"}, {"send_bps": 0, "recv_bps": 0, "name":
"vif1"}, {"send_bps": 9376, "recv_bps": 19312, "name": "c0b"}, {"send_bps":
0, "recv_bps": 0, "name": "e0M"}, {"send_bps": 0, "recv_bps": 0, "name":
"losk"}]}
I did not have to change my schema.
For anyone interested, I verified I could write my array of records with
the schema using this test program.
public class ArrayOfRecTest {
public static void main(String[] args) throws IOException {
String fileName = "/tmp/test.avro";
File avroFile = new File(fileName);
// NOTE: MyRecord.java was generated from MyRecord.avsc
and avro.compiler.specific.SchemaTask
Schema schema = new MyRecord().getSchema();
DatumWriter<MyRecord> writer = new SpecificDatumWriter<MyRecord>(schema);
DataFileWriter<MyRecord> dataFileWriter = new
DataFileWriter<MyRecord>(writer);
dataFileWriter.create(schema, avroFile);
MyRecord record = new MyRecord();
record.setHostname("localhost");
record.setTimeStamp(new Long(1357639723));
List<Ifnet> ifnetList = new ArrayList<Ifnet>();
Ifnet ifc1 = new Ifnet("eth0", 1234L, 5678L); ifnetList.add(ifc1);
Ifnet ifc2 = new Ifnet("eth1", 100L, 200L); ifnetList.add(ifc2);
Ifnet ifc3 = new Ifnet("eth2", 0L, 0L); ifnetList.add(ifc3);
record.setIfnet(ifnetList);
dataFileWriter.append(record);
dataFileWriter.flush();
dataFileWriter.close();
}
}
Alan
On Tue, Jan 8, 2013 at 11:36 AM, Scott Carey <[email protected]> wrote:
>
> On 1/7/13 8:35 AM, "Alan Miller" <[email protected]> wrote:
>
>
> >Hi, I have a schema with an array of records (I'm open to other
> >suggestions too) field
> >called ifnet to store misc attribute name/values for a host's network
> >interfaces.
> >e.g.
> >
> >
> >{ "type": "record",
> > "namespace": "com.company.avro.data",
> > "name": "MyRecord",
> > "doc": "My Data Record.",
> > "fields": [
> > // (required) fields
> > {"name": "time_stamp", "type": "long"
> > },
> > {"name": "hostname", "type": "string"
> > },
> >
> > // (optional) array of ifnet instances
> > {"name": "ifnet",
> > "type": ["null", {
> > "type": "array",
> > "items": { "type": "record", "name": "Ifnet",
> > "namespace": "com.company.avro.data",
> > "fields": [ {"name": "name", "type":
> >"string"},
> > {"name": "send_bps", "type":
> >"long" },
> > {"name": "recv_bps", "type":
> >"long" }
> > ]
> > }
> > }
> > ]
> > }
> >
> > ]
> >}
>
> First thought: Why the union of null and the array? It may be easier to
> simply have an empty list when there are no Ifnet data.
>
>
>
> >
> >
> >I can write the records, (time_stamp and hostname are correct) but
> >my "array of records" field (ifnet) only contains the last element of my
> >java List.
> >
> >Am I writing the field correctly? I'm trying to write the ifnet field
> >with a
> >java.util.List<com.company.avro.data.Ifnet>
> >
> >Here's the related code lines that write the ifnet field. (Yes, I'm
> >attempting to use reflection
> >because Ifnet is only 1 of approx 11 other array of record fields I'm
> >trying to implement.)
> >
> > Class[] paramObj = new Class[1];
> > paramObj[0] = Ifnet.class;
> > Method method = cls.getMethod(methodName, List.class);
> > jsonObj = new Ifnet();
> > listOfObj = new ArrayList<Ifnet>();
> > .......
> >
> >
> > // in a loop building the List<Ifnet>.......
> >
> > LOG.info(String.format(" [%s] %s %s(%s) as %s", name,
> >k,methNm,v,types[j].toString()));
> > .......
> >
> > LOG.info(String.format(" [%s] setting name to %s", name, name));
> >
> > .......
> >
> > istOfObj.add(jsonObj);
> >
> > .......
> >
> > // then finally I call invoke with a List of Ifnet records
> >
> > if (method != null) { method.invoke(obj, listOfObj); }
> > LOG.info(String.format(" invoking %s.%s",
> >method.getClass().getSimpleName(), method.getName()));
> > LOG.info(String.format(" param: listObj<%s> with %d entries" ,
> >jsonObj.getClass().getName(), listOfObj.size()));
> >
> >
> >and the respective output
> >20130107T172303 INFO c.c.a.d.MyDriver - Set ifnet
> >json via setIfnet(Ifnet object)
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0c] setting name to e0c
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0c] send_bps setSendBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0c] setting name to e0c
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0c] recv_bps setRecvBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0d] setting name to e0d
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0d] send_bps setSendBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0d] setting name to e0d
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0d] recv_bps setRecvBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0a] setting name to e0a
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0a] send_bps
> >setSendBps(170720) as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0a] setting name to e0a
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0a] recv_bps
> >setRecvBps(244480) as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0b] setting name to e0b
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0b] send_bps setSendBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0b] setting name to e0b
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0b] recv_bps setRecvBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0P] setting name to e0P
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0P] send_bps setSendBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0P] setting name to e0P
> >20130107T172303 INFO c.c.a.d.MyDriver - [e0P] recv_bps setRecvBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [losk] setting name to losk
> >20130107T172303 INFO c.c.a.d.MyDriver - [losk] send_bps setSendBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - [losk] setting name to losk
> >20130107T172303 INFO c.c.a.d.MyDriver - [losk] recv_bps setRecvBps(0)
> >as class java.lang.Long
> >20130107T172303 INFO c.c.a.d.MyDriver - invoking Method.setIfnet
> >20130107T172303 INFO c.c.a.d.MyDriver - param:
> >listObj<com.synopsys.iims.be.storage.Ifnet> with 6 entries
> >
> >20130107T172303 INFO c.c.a.d.MyDriver - Set time_stamp
> >integer via setTimeStamp to 1357513251
> >
> >
> >When I dump the records I see an array of 6 entries but the values all
> >reflect the last last entry in my java.util.List.
> >The print statements suggest I'm passing a list of 6 items with different
> >values,
> >
> > [amiller@localhost $] hadoop fs -copyToLocal /data/test_2013-01-07.avro
> > [amiller@localhost $] avro cat --fields=time_stamp,ifnet
> >test_2013-01-07.avro|less
> > {"time_stamp": 1357513251, "ifnet": [{"send_bps": 0, "recv_bps": 0,
> >"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
> >{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0,
> >"recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
> > 0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
> >
> > {"time_stamp": 1357513550, "ifnet": [{"send_bps": 0, "recv_bps": 0,
> >"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
> >{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0,
> >"recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
> > 0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
> >
> >
> >
> >Any ideas.
> >
>
> What API are you using? Generic, Specific, or Reflect? I am suspicious
> of the reflection, does it work when you bind it to one specific type? I
> sus=
> pect you could use generics to avoid the reflection but I have seen
> similar tasks get a little ugly.
>
> This is something you should be able to step through in a debugger; when
> you append the datum to the file, you can check the state of the object
> being written and that it has all of the expected data, then step through
> the code in the DatumWriter (if you use an IDE and maven, it should be
> trivial to attach the Avro source to your debugger) and see what it does
> with your list.
>
>
>
>
> >
> >
> >
> >
> >Alan
>
>
>
