[jira] [Updated] (AVRO-1065) NodeRecord::isValid() treats records with no fields as invalid
[
https://issues.apache.org/jira/browse/AVRO-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1065:
--
Attachment: AVRO-1065.patch
This patch fixes the problem.
> NodeRecord::isValid() treats records with no fields as invalid
> --
>
> Key: AVRO-1065
> URL: https://issues.apache.org/jira/browse/AVRO-1065
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.6.3
>Reporter: Keh-Li Sheng
> Attachments: AVRO-1065.patch
>
>
> Implementation of NodeRecord in NodeImpl.hh requires that a record have at
> least 1 field in it but this is not required as per the specification at
> http://avro.apache.org/docs/1.6.3/spec.html#schema_record. Either remove the
> condition or update the spec. This only seems to be a quirk in c++.
> {code:title=NodeImpl.hh}
> class AVRO_DECL NodeRecord : public NodeImplRecord
> {
> ...
> bool isValid() const {
> return (
> (nameAttribute_.size() == 1) &&
> (leafAttributes_.size() > 0) &&
> (leafAttributes_.size() == leafNameAttributes_.size())
>);
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1070) AvroSequenceFileOutputFormat is in wrong package.
[
https://issues.apache.org/jira/browse/AVRO-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1070:
--
Attachment: AVRO-1070.patch
I changed the package name in the Java source because the change is minimal.
Maybe moving it to the {{io}} directory instead is more appropriate, I don't
know.
> AvroSequenceFileOutputFormat is in wrong package.
> -
>
> Key: AVRO-1070
> URL: https://issues.apache.org/jira/browse/AVRO-1070
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Thiruvalluvan M. G.
>Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-1070.patch
>
>
> The source directory {{org/apache/avro/mapreduce}} and package name
> {{org.apache.avro.hadoop.io}} do not match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1070) AvroSequenceFileOutputFormat is in wrong package.
Thiruvalluvan M. G. created AVRO-1070:
-
Summary: AvroSequenceFileOutputFormat is in wrong package.
Key: AVRO-1070
URL: https://issues.apache.org/jira/browse/AVRO-1070
Project: Avro
Issue Type: Bug
Components: java
Reporter: Thiruvalluvan M. G.
Assignee: Thiruvalluvan M. G.
The source directory {{org/apache/avro/mapreduce}} and package name
{{org.apache.avro.hadoop.io}} do not match.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1068) Avro Java does not build on recent Eclipse versions with m2eclipse
[ https://issues.apache.org/jira/browse/AVRO-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-1068: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1328865. Thanks Scott for reviewing it. > Avro Java does not build on recent Eclipse versions with m2eclipse > -- > > Key: AVRO-1068 > URL: https://issues.apache.org/jira/browse/AVRO-1068 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.7.0 > > Attachments: AVRO-1068.patch > > > It is so because, in the recent versions of m2eclipse require manual mapping > of non-standard maven lifecycle tasks. The forthcoming fix addresses that > problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-954) C++ - Typo in JsonCodec.cc
[ https://issues.apache.org/jira/browse/AVRO-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-954: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1328833. After adjusting for the recent changes in change in location of Json code. Thank you Nebojsa Sabovic > C++ - Typo in JsonCodec.cc > -- > > Key: AVRO-954 > URL: https://issues.apache.org/jira/browse/AVRO-954 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.6.0 >Reporter: Nebojsa Sabovic >Assignee: Nebojsa Sabovic >Priority: Minor > Labels: c++ > Attachments: AVRO-954.patch > > > JsonCodec.cc has a line that reads: if (isdigit(ch) || '-') > This doesn't do anything useful, we should change it to read: if (isdigit(ch) > || ch == '-') -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1066) ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph
[
https://issues.apache.org/jira/browse/AVRO-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258897#comment-13258897
]
Thiruvalluvan M. G. commented on AVRO-1066:
---
I'll commit this on Tuesday if there is no objection.
> ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json
> encoder to serialize a deep object graph
> ---
>
> Key: AVRO-1066
> URL: https://issues.apache.org/jira/browse/AVRO-1066
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.6.3
>Reporter: Daniel Lord
>Assignee: Thiruvalluvan M. G.
> Fix For: 1.7.0
>
> Attachments: AVRO-1066.patch
>
>
> I have an avro record that can have an array of children records of the same
> type. If this structure gets to be too deep then I continually get an
> ArrayOutOfBoundsException in ParsingEncoder when trying to use a json
> encoder. This works fine when using a binary encoder. My schema looks
> something like this:
> {
> "name" : "MyRecord",
> "type" : "record",
> "fields" : [
> {
> "name" : "fooField",
> "type" : "int"
> },
> {
> "name" : "childRecords",
> "type" : [ "null", { "type" : "array", "items" : "MyRecord" } ]
> }
> ]
> }
> The code I'm using to capture a JSON string for debugging looks like this:
> ByteArrayOutputStream os = null;
> try {
> os = new ByteArrayOutputStream(2048);
> final Encoder jsonEncoder =
> EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os);
> final DatumWriter datumWriter = new
> SpecificDatumWriter(MyRecord.class);
> datumWriter.write(record, jsonEncoder);
> jsonEncoder.flush();
> return new String(os.toByteArray(), Charset.defaultCharset());
> } catch (IOException e) {
> return null;
> } finally {
> if (os != null) {
> try {
> os.close();
> } catch (IOException e) {
> }
> }
> }
> The error I get looks like this:
> java.lang.ArrayIndexOutOfBoundsException: 10
> at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
> at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> org.apache.avro.generic.GenericDatumW
[jira] [Commented] (AVRO-1065) NodeRecord::isValid() treats records with no fields as invalid
[
https://issues.apache.org/jira/browse/AVRO-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258886#comment-13258886
]
Thiruvalluvan M. G. commented on AVRO-1065:
---
+1.
> NodeRecord::isValid() treats records with no fields as invalid
> --
>
> Key: AVRO-1065
> URL: https://issues.apache.org/jira/browse/AVRO-1065
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.6.3
>Reporter: Keh-Li Sheng
>
> Implementation of NodeRecord in NodeImpl.hh requires that a record have at
> least 1 field in it but this is not required as per the specification at
> http://avro.apache.org/docs/1.6.3/spec.html#schema_record. Either remove the
> condition or update the spec. This only seems to be a quirk in c++.
> {code:title=NodeImpl.hh}
> class AVRO_DECL NodeRecord : public NodeImplRecord
> {
> ...
> bool isValid() const {
> return (
> (nameAttribute_.size() == 1) &&
> (leafAttributes_.size() > 0) &&
> (leafAttributes_.size() == leafNameAttributes_.size())
>);
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-869) Lifetimes of streams and encoder/decoders not managed propertly
Lifetimes of streams and encoder/decoders not managed propertly --- Key: AVRO-869 URL: https://issues.apache.org/jira/browse/AVRO-869 Project: Avro Issue Type: Bug Components: c++ Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Encoders and decoders are designed to be used with a single OutputStream and InputStream respectively. Init() dunction is used to attach a new OutputStream/InputStream with Encoder/Decoder in place of the current one. There is no way to cleanly detach the current stream. The init method cleans up the current stream before detaching it. If the old stream is not valid, it tries to clean up and crash. The forthcoming patch will fix this issue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-806) add a column-major codec for data files
[ https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-806: - Attachment: avro-file-columnar.pdf AVRO-806-v2.patch Here is a patch which implements the same idea, more formally. Please see the attached file that Raymie Stata wrote about the approach we are proposing. Some comments on the patch: * The implementation basically works. Further optimizations are possible. One pending work is the optimization to avoid looking into columns that are needed. At present the traditional schema resolution works, but if the reader's and writer's schemas are such that certain column can be completely skipped, they are not skipped. * Data file writer uses a trick that in case of exceptions, it flushes things up to the previous record. That trick won't work with columnar storage because it can be flushed just once per block. (I've commented out the test case for that.) * Raymie and I have designed the solution in layers - one layer that is capable of writing and reading columnar store. The column to data item assignment are a separate layer. We have a default that assigns each item to a separate column up to a depth. But the users can supply their own custom column assignments. Yet another layer is the ability to put the columns in a block. In case someone wants to use file-based columnar storage, they can do so easily on top the first two layers. * For now, unions have a single column, irrespective of the number and type of branches. * I don't think extending the codec mechanism for supporting columnar store will work. Columnar store is orthogonal to codecs. Codecs are about storing blocks with compression. Decoders decide how the contents should be interpreted. I think, the way to support columnar storage is to replace binary encoder/decoder with columnar encoder/decoders. I've demonstrated in the Data file reader and writer. In order to support both binary encoder/decoder and columnar encoder decoder, I pushes bytesBuffered() to Encoder from BinaryEncoder and isEnd() to Decoder from BinaryDecoder. I don't think these changes will break anything. * The implementation is not complete, We have to let the user of data file writer choose columnar storage instead of binarystorage. I've not implemeted it yet. * One test in FileSpanStorage is failing possibly because, I think, it assumes something about the way data file stores using binary encoder. I'm not sure. * If we make Data file writer/reader handle different encoders/decoders, then changes to columnar storage is reasonably well-isolated. * There are a couple of unrelated changes (in tools) that were required to make the tests pass on my machine. Please ignore them. > add a column-major codec for data files > --- > > Key: AVRO-806 > URL: https://issues.apache.org/jira/browse/AVRO-806 > Project: Avro > Issue Type: New Feature > Components: java, spec >Reporter: Doug Cutting >Assignee: Doug Cutting > Attachments: AVRO-806-v2.patch, AVRO-806.patch, avro-file-columnar.pdf > > > Define a codec that, when a data file's schema is a record schema, writes > blocks within the file in column-major order. This would permit better > compression and also permit efficient skipping of fields that are not of > interest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-810) Add strong naming to C# assemblies
[ https://issues.apache.org/jira/browse/AVRO-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040753#comment-13040753 ] Thiruvalluvan M. G. commented on AVRO-810: -- +1 This works for on Linux without trouble. On Windows it gives an error during patch, but if you ignore that it goes through. Is there a reason why the strong naming is not applied for codegen (/Avro.codegen.csproj)? > Add strong naming to C# assemblies > -- > > Key: AVRO-810 > URL: https://issues.apache.org/jira/browse/AVRO-810 > Project: Avro > Issue Type: Improvement > Components: csharp >Reporter: Eric Hauser >Priority: Minor > Attachments: AVRO-810.patch, Avro.snk > > > The C# assemblies should be strong named. A .gitignore file for common end > user Visual Studio files is also in the patch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-825) C++ code generation issues with records when member of the record is of type itself
[
https://issues.apache.org/jira/browse/AVRO-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-825:
-
Attachment: AVRO-825-v5.patch
Ramana and I had a few e-mail exchanges where he found some corner cases where
the code generator did not produce correct code. This one fixes those problems
and had additional test cases.
> C++ code generation issues with records when member of the record is of type
> itself
> ---
>
> Key: AVRO-825
> URL: https://issues.apache.org/jira/browse/AVRO-825
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.5.1
>Reporter: Ramana Suvarapu
>Assignee: Thiruvalluvan M. G.
>Priority: Blocker
> Attachments: AVRO-825-v2.patch, AVRO-825-v5.patch, AVRO-825.patch
>
>
> I am trying to generate C++ code for the following schema using new
> avrogencpp. This schema is one the of the example given in Avro specification
> {
> "type": "record",
> "name": "LongList",
> "aliases": ["LinkedLongs"], // old name for this
> "fields" : [
> {"name": "value", "type": "long"}, // each element has a long
> {"name": "next", "type": ["LongList", "null"]} // optional next element
> ]
> }
> It generated the following .hh file and when I tried to compile I am getting
> #ifndef __TEST_LINKEDLIST_HH_1817083040__H_
> #define __TEST_LINKEDLIST_HH_1817083040__H_
> #include "boost/any.hpp"
> #include "Specific.hh"
> #include "Encoder.hh"
> #include "Decoder.hh"
> struct _imaginary_avr_Union__0__ {
> private:
> size_t idx_;
> boost::any value_;
> public:
> size_t idx() const { return idx_; }
> LongList get_LongList() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union");
> }
> return boost::any_cast(value_);
> }
> void set_LongList(const LongList& v) {
> idx_ = 0;
> value_ = v;
> }
> void set_null() {
> idx_ = 1;
> value_ = boost::any();
> }
> _imaginary_avr_Union__0__() : idx_(0) {
> value_ = LongList();
> }
> };
> struct LongList {
> int64_t value;
> _imaginary_avr_Union__0__ next;
> };
> namespace avro {
> template<> struct codec_traits<_imaginary_avr_Union__0__> {
> static void encode(Encoder& e, _imaginary_avr_Union__0__ v) {
> e.encodeUnionIndex(v.idx());
> switch (v.idx()) {
> case 0:
> avro::encode(e, v.get_LongList());
> break;
> case 1:
> e.encodeNull();
> break;
> }
> }
> static void decode(Decoder& d, _imaginary_avr_Union__0__& v) {
> size_t n = d.decodeUnionIndex();
> if (n >= 2) { throw avro::Exception("Union index too big"); }
> switch (n) {
> case 0:
> {
> LongList vv;
> avro::decode(d, vv);
> v.set_LongList(vv);
> }
> break;
> case 1:
> d.decodeNull();
> v.set_null();
> break;
> }
> }
> };
> template<> struct codec_traits {
> static void encode(Encoder& e, const LongList& v) {
> avro::encode(e, v.value);
> avro::encode(e, v.next);
> }
> static void decode(Decoder& d, LongList& v) {
> avro::decode(d, v.value);
> avro::decode(d, v.next);
> }
> };
> }
> #endif
> When I tried to compile I am getting following errors due to incomplete type.
> Errors
> In file included from
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/AvrogencppTests.cc:24:
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:34: error:
> `LongList' does not name a type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: expected
> `,' or `...' before '&' token
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: ISO C++
> forbids declaration of `LongList' with no type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In member function
> `void _imaginary_avr_Union__0__::set_LongList(int)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:42: error: `v' was
> not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In constructor
> `_imaginary_avr_Union__0__::_imaginary_avr_Union__0__()':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:49: error:
> `LongList' was not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::encode(avro::Encoder&,
> _imaginary_avr_Union__0__)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:64: error: 'struct
> _imaginary_avr_Union__0
[jira] [Updated] (AVRO-825) C++ code generation issues with records when member of the record is of type itself
[
https://issues.apache.org/jira/browse/AVRO-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-825:
-
Attachment: AVRO-825-v2.patch
This patch addresses a few more requirements for forward declarations
> C++ code generation issues with records when member of the record is of type
> itself
> ---
>
> Key: AVRO-825
> URL: https://issues.apache.org/jira/browse/AVRO-825
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.5.1
>Reporter: Ramana Suvarapu
>Assignee: Thiruvalluvan M. G.
>Priority: Blocker
> Attachments: AVRO-825-v2.patch, AVRO-825.patch
>
>
> I am trying to generate C++ code for the following schema using new
> avrogencpp. This schema is one the of the example given in Avro specification
> {
> "type": "record",
> "name": "LongList",
> "aliases": ["LinkedLongs"], // old name for this
> "fields" : [
> {"name": "value", "type": "long"}, // each element has a long
> {"name": "next", "type": ["LongList", "null"]} // optional next element
> ]
> }
> It generated the following .hh file and when I tried to compile I am getting
> #ifndef __TEST_LINKEDLIST_HH_1817083040__H_
> #define __TEST_LINKEDLIST_HH_1817083040__H_
> #include "boost/any.hpp"
> #include "Specific.hh"
> #include "Encoder.hh"
> #include "Decoder.hh"
> struct _imaginary_avr_Union__0__ {
> private:
> size_t idx_;
> boost::any value_;
> public:
> size_t idx() const { return idx_; }
> LongList get_LongList() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union");
> }
> return boost::any_cast(value_);
> }
> void set_LongList(const LongList& v) {
> idx_ = 0;
> value_ = v;
> }
> void set_null() {
> idx_ = 1;
> value_ = boost::any();
> }
> _imaginary_avr_Union__0__() : idx_(0) {
> value_ = LongList();
> }
> };
> struct LongList {
> int64_t value;
> _imaginary_avr_Union__0__ next;
> };
> namespace avro {
> template<> struct codec_traits<_imaginary_avr_Union__0__> {
> static void encode(Encoder& e, _imaginary_avr_Union__0__ v) {
> e.encodeUnionIndex(v.idx());
> switch (v.idx()) {
> case 0:
> avro::encode(e, v.get_LongList());
> break;
> case 1:
> e.encodeNull();
> break;
> }
> }
> static void decode(Decoder& d, _imaginary_avr_Union__0__& v) {
> size_t n = d.decodeUnionIndex();
> if (n >= 2) { throw avro::Exception("Union index too big"); }
> switch (n) {
> case 0:
> {
> LongList vv;
> avro::decode(d, vv);
> v.set_LongList(vv);
> }
> break;
> case 1:
> d.decodeNull();
> v.set_null();
> break;
> }
> }
> };
> template<> struct codec_traits {
> static void encode(Encoder& e, const LongList& v) {
> avro::encode(e, v.value);
> avro::encode(e, v.next);
> }
> static void decode(Decoder& d, LongList& v) {
> avro::decode(d, v.value);
> avro::decode(d, v.next);
> }
> };
> }
> #endif
> When I tried to compile I am getting following errors due to incomplete type.
> Errors
> In file included from
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/AvrogencppTests.cc:24:
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:34: error:
> `LongList' does not name a type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: expected
> `,' or `...' before '&' token
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: ISO C++
> forbids declaration of `LongList' with no type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In member function
> `void _imaginary_avr_Union__0__::set_LongList(int)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:42: error: `v' was
> not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In constructor
> `_imaginary_avr_Union__0__::_imaginary_avr_Union__0__()':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:49: error:
> `LongList' was not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::encode(avro::Encoder&,
> _imaginary_avr_Union__0__)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:64: error: 'struct
> _imaginary_avr_Union__0__' has no member named 'get_LongList'
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static voi
[jira] [Updated] (AVRO-825) C++ code generation issues with records when member of the record is of type itself
[
https://issues.apache.org/jira/browse/AVRO-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-825:
-
Status: Patch Available (was: Open)
> C++ code generation issues with records when member of the record is of type
> itself
> ---
>
> Key: AVRO-825
> URL: https://issues.apache.org/jira/browse/AVRO-825
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.5.1
>Reporter: Ramana Suvarapu
>Assignee: Thiruvalluvan M. G.
>Priority: Blocker
> Attachments: AVRO-825.patch
>
>
> I am trying to generate C++ code for the following schema using new
> avrogencpp. This schema is one the of the example given in Avro specification
> {
> "type": "record",
> "name": "LongList",
> "aliases": ["LinkedLongs"], // old name for this
> "fields" : [
> {"name": "value", "type": "long"}, // each element has a long
> {"name": "next", "type": ["LongList", "null"]} // optional next element
> ]
> }
> It generated the following .hh file and when I tried to compile I am getting
> #ifndef __TEST_LINKEDLIST_HH_1817083040__H_
> #define __TEST_LINKEDLIST_HH_1817083040__H_
> #include "boost/any.hpp"
> #include "Specific.hh"
> #include "Encoder.hh"
> #include "Decoder.hh"
> struct _imaginary_avr_Union__0__ {
> private:
> size_t idx_;
> boost::any value_;
> public:
> size_t idx() const { return idx_; }
> LongList get_LongList() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union");
> }
> return boost::any_cast(value_);
> }
> void set_LongList(const LongList& v) {
> idx_ = 0;
> value_ = v;
> }
> void set_null() {
> idx_ = 1;
> value_ = boost::any();
> }
> _imaginary_avr_Union__0__() : idx_(0) {
> value_ = LongList();
> }
> };
> struct LongList {
> int64_t value;
> _imaginary_avr_Union__0__ next;
> };
> namespace avro {
> template<> struct codec_traits<_imaginary_avr_Union__0__> {
> static void encode(Encoder& e, _imaginary_avr_Union__0__ v) {
> e.encodeUnionIndex(v.idx());
> switch (v.idx()) {
> case 0:
> avro::encode(e, v.get_LongList());
> break;
> case 1:
> e.encodeNull();
> break;
> }
> }
> static void decode(Decoder& d, _imaginary_avr_Union__0__& v) {
> size_t n = d.decodeUnionIndex();
> if (n >= 2) { throw avro::Exception("Union index too big"); }
> switch (n) {
> case 0:
> {
> LongList vv;
> avro::decode(d, vv);
> v.set_LongList(vv);
> }
> break;
> case 1:
> d.decodeNull();
> v.set_null();
> break;
> }
> }
> };
> template<> struct codec_traits {
> static void encode(Encoder& e, const LongList& v) {
> avro::encode(e, v.value);
> avro::encode(e, v.next);
> }
> static void decode(Decoder& d, LongList& v) {
> avro::decode(d, v.value);
> avro::decode(d, v.next);
> }
> };
> }
> #endif
> When I tried to compile I am getting following errors due to incomplete type.
> Errors
> In file included from
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/AvrogencppTests.cc:24:
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:34: error:
> `LongList' does not name a type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: expected
> `,' or `...' before '&' token
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: ISO C++
> forbids declaration of `LongList' with no type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In member function
> `void _imaginary_avr_Union__0__::set_LongList(int)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:42: error: `v' was
> not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In constructor
> `_imaginary_avr_Union__0__::_imaginary_avr_Union__0__()':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:49: error:
> `LongList' was not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::encode(avro::Encoder&,
> _imaginary_avr_Union__0__)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:64: error: 'struct
> _imaginary_avr_Union__0__' has no member named 'get_LongList'
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::decode(avro::Decoder&,
> _imag
[jira] [Updated] (AVRO-825) C++ code generation issues with records when member of the record is of type itself
[
https://issues.apache.org/jira/browse/AVRO-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-825:
-
Attachment: AVRO-825.patch
The attached patch has a test for this kind of schemas and fixes the bug.
> C++ code generation issues with records when member of the record is of type
> itself
> ---
>
> Key: AVRO-825
> URL: https://issues.apache.org/jira/browse/AVRO-825
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.5.1
>Reporter: Ramana Suvarapu
>Assignee: Thiruvalluvan M. G.
>Priority: Blocker
> Attachments: AVRO-825.patch
>
>
> I am trying to generate C++ code for the following schema using new
> avrogencpp. This schema is one the of the example given in Avro specification
> {
> "type": "record",
> "name": "LongList",
> "aliases": ["LinkedLongs"], // old name for this
> "fields" : [
> {"name": "value", "type": "long"}, // each element has a long
> {"name": "next", "type": ["LongList", "null"]} // optional next element
> ]
> }
> It generated the following .hh file and when I tried to compile I am getting
> #ifndef __TEST_LINKEDLIST_HH_1817083040__H_
> #define __TEST_LINKEDLIST_HH_1817083040__H_
> #include "boost/any.hpp"
> #include "Specific.hh"
> #include "Encoder.hh"
> #include "Decoder.hh"
> struct _imaginary_avr_Union__0__ {
> private:
> size_t idx_;
> boost::any value_;
> public:
> size_t idx() const { return idx_; }
> LongList get_LongList() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union");
> }
> return boost::any_cast(value_);
> }
> void set_LongList(const LongList& v) {
> idx_ = 0;
> value_ = v;
> }
> void set_null() {
> idx_ = 1;
> value_ = boost::any();
> }
> _imaginary_avr_Union__0__() : idx_(0) {
> value_ = LongList();
> }
> };
> struct LongList {
> int64_t value;
> _imaginary_avr_Union__0__ next;
> };
> namespace avro {
> template<> struct codec_traits<_imaginary_avr_Union__0__> {
> static void encode(Encoder& e, _imaginary_avr_Union__0__ v) {
> e.encodeUnionIndex(v.idx());
> switch (v.idx()) {
> case 0:
> avro::encode(e, v.get_LongList());
> break;
> case 1:
> e.encodeNull();
> break;
> }
> }
> static void decode(Decoder& d, _imaginary_avr_Union__0__& v) {
> size_t n = d.decodeUnionIndex();
> if (n >= 2) { throw avro::Exception("Union index too big"); }
> switch (n) {
> case 0:
> {
> LongList vv;
> avro::decode(d, vv);
> v.set_LongList(vv);
> }
> break;
> case 1:
> d.decodeNull();
> v.set_null();
> break;
> }
> }
> };
> template<> struct codec_traits {
> static void encode(Encoder& e, const LongList& v) {
> avro::encode(e, v.value);
> avro::encode(e, v.next);
> }
> static void decode(Decoder& d, LongList& v) {
> avro::decode(d, v.value);
> avro::decode(d, v.next);
> }
> };
> }
> #endif
> When I tried to compile I am getting following errors due to incomplete type.
> Errors
> In file included from
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/AvrogencppTests.cc:24:
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:34: error:
> `LongList' does not name a type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: expected
> `,' or `...' before '&' token
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: ISO C++
> forbids declaration of `LongList' with no type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In member function
> `void _imaginary_avr_Union__0__::set_LongList(int)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:42: error: `v' was
> not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In constructor
> `_imaginary_avr_Union__0__::_imaginary_avr_Union__0__()':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:49: error:
> `LongList' was not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::encode(avro::Encoder&,
> _imaginary_avr_Union__0__)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:64: error: 'struct
> _imaginary_avr_Union__0__' has no member named 'get_LongList'
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_t
[jira] [Assigned] (AVRO-825) C++ code generation issues with records when member of the record is of type itself
[
https://issues.apache.org/jira/browse/AVRO-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. reassigned AVRO-825:
Assignee: Thiruvalluvan M. G.
> C++ code generation issues with records when member of the record is of type
> itself
> ---
>
> Key: AVRO-825
> URL: https://issues.apache.org/jira/browse/AVRO-825
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.5.1
>Reporter: Ramana Suvarapu
>Assignee: Thiruvalluvan M. G.
>Priority: Blocker
>
> I am trying to generate C++ code for the following schema using new
> avrogencpp. This schema is one the of the example given in Avro specification
> {
> "type": "record",
> "name": "LongList",
> "aliases": ["LinkedLongs"], // old name for this
> "fields" : [
> {"name": "value", "type": "long"}, // each element has a long
> {"name": "next", "type": ["LongList", "null"]} // optional next element
> ]
> }
> It generated the following .hh file and when I tried to compile I am getting
> #ifndef __TEST_LINKEDLIST_HH_1817083040__H_
> #define __TEST_LINKEDLIST_HH_1817083040__H_
> #include "boost/any.hpp"
> #include "Specific.hh"
> #include "Encoder.hh"
> #include "Decoder.hh"
> struct _imaginary_avr_Union__0__ {
> private:
> size_t idx_;
> boost::any value_;
> public:
> size_t idx() const { return idx_; }
> LongList get_LongList() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union");
> }
> return boost::any_cast(value_);
> }
> void set_LongList(const LongList& v) {
> idx_ = 0;
> value_ = v;
> }
> void set_null() {
> idx_ = 1;
> value_ = boost::any();
> }
> _imaginary_avr_Union__0__() : idx_(0) {
> value_ = LongList();
> }
> };
> struct LongList {
> int64_t value;
> _imaginary_avr_Union__0__ next;
> };
> namespace avro {
> template<> struct codec_traits<_imaginary_avr_Union__0__> {
> static void encode(Encoder& e, _imaginary_avr_Union__0__ v) {
> e.encodeUnionIndex(v.idx());
> switch (v.idx()) {
> case 0:
> avro::encode(e, v.get_LongList());
> break;
> case 1:
> e.encodeNull();
> break;
> }
> }
> static void decode(Decoder& d, _imaginary_avr_Union__0__& v) {
> size_t n = d.decodeUnionIndex();
> if (n >= 2) { throw avro::Exception("Union index too big"); }
> switch (n) {
> case 0:
> {
> LongList vv;
> avro::decode(d, vv);
> v.set_LongList(vv);
> }
> break;
> case 1:
> d.decodeNull();
> v.set_null();
> break;
> }
> }
> };
> template<> struct codec_traits {
> static void encode(Encoder& e, const LongList& v) {
> avro::encode(e, v.value);
> avro::encode(e, v.next);
> }
> static void decode(Decoder& d, LongList& v) {
> avro::decode(d, v.value);
> avro::decode(d, v.next);
> }
> };
> }
> #endif
> When I tried to compile I am getting following errors due to incomplete type.
> Errors
> In file included from
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/AvrogencppTests.cc:24:
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:34: error:
> `LongList' does not name a type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: expected
> `,' or `...' before '&' token
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:40: error: ISO C++
> forbids declaration of `LongList' with no type
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In member function
> `void _imaginary_avr_Union__0__::set_LongList(int)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:42: error: `v' was
> not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In constructor
> `_imaginary_avr_Union__0__::_imaginary_avr_Union__0__()':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:49: error:
> `LongList' was not declared in this scope
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::encode(avro::Encoder&,
> _imaginary_avr_Union__0__)':
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh:64: error: 'struct
> _imaginary_avr_Union__0__' has no member named 'get_LongList'
> /home/rsuvarap/avro-src-1.5.1/lang/c++/test/linkedlist.hh: In static member
> function `static void
> avro::codec_traits<_imaginary_avr_Union__0__>::decode(avro::Decoder&,
> _imaginary_avr_Union__0__&)':
> /home/rsuvarap
[jira] [Assigned] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-792: Assignee: (was: Thiruvalluvan M. G.) I don't have a theory that can explain this behavior. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792-2.patch, AVRO-792-3.patch, AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrappe
[jira] [Updated] (AVRO-793) A strange problem when I am trying to read avro record with a subset of the schema.
[
https://issues.apache.org/jira/browse/AVRO-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-793:
-
Attachment: AVRO-793-test.patch
AVRO-793.patch
Very subtle bug. If there is an array needs to be skipped and that happens to
be the last field of a record, and the record is contained in an outer array,
it does not get skipped properly.
The test patch has the test that catches the bug and the main patch has the
solution.
> A strange problem when I am trying to read avro record with a subset of the
> schema.
> ---
>
> Key: AVRO-793
> URL: https://issues.apache.org/jira/browse/AVRO-793
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.5.0
> Environment: Avro1.5,Windows xp/Ubuntu 10.0.4
>Reporter: Yingzhong Xu
>Assignee: Thiruvalluvan M. G.
>Priority: Critical
> Labels: Avro, Reading, Schema, Write
> Fix For: 1.5.1
>
> Attachments: AVRO-793-test.patch, AVRO-793.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hi, all. When I am trying to read avro file with a subset of that
> schema(because I do not need all the details).I meet a strange problem.
> 1.I write data using this schema:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> },
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> 2.Here is a JSONObject for that schema.
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "device":"WMA1",
> "children":["WMB1","WMB2"]
> },
> {
> "device":"WMA2",
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 3.I write that record succefully.And it is okay if I use this schema for
> reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> the result is :
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "children":["WMB1","WMB2"]
> },
> {
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 4.But if i want to igonre the "children" part instead of "device", I use
> this schema for reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> }
> ]
> }
> }
> }
> }
> ]
> }
> Unfortunately,I get exception:
> java.lang.ArrayIndexOutOfBoundsException: -8
> cause:java.lang.ArrayIndexOutOfBoundsException
> at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:122)
> at org.apache.avro.io.BinaryDecoder.skipString(BinaryDecoder.java:262)
> at org.apache.avro.io.ValidatingDecoder.skipString(ValidatingDecoder.java:113)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:60)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipRepeater(SkipParser.java:83)
> at org.apache.avro.io.ValidatingDecoder.skipArray(ValidatingDecoder.java:195)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:70)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipSymbol(SkipParser.java:93)
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:226)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at
> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:162)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatu
[jira] [Updated] (AVRO-793) A strange problem when I am trying to read avro record with a subset of the schema.
[
https://issues.apache.org/jira/browse/AVRO-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-793:
-
Status: Patch Available (was: Open)
> A strange problem when I am trying to read avro record with a subset of the
> schema.
> ---
>
> Key: AVRO-793
> URL: https://issues.apache.org/jira/browse/AVRO-793
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.5.0
> Environment: Avro1.5,Windows xp/Ubuntu 10.0.4
>Reporter: Yingzhong Xu
>Assignee: Thiruvalluvan M. G.
>Priority: Critical
> Labels: Avro, Reading, Schema, Write
> Fix For: 1.5.1
>
> Attachments: AVRO-793-test.patch, AVRO-793.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hi, all. When I am trying to read avro file with a subset of that
> schema(because I do not need all the details).I meet a strange problem.
> 1.I write data using this schema:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> },
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> 2.Here is a JSONObject for that schema.
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "device":"WMA1",
> "children":["WMB1","WMB2"]
> },
> {
> "device":"WMA2",
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 3.I write that record succefully.And it is okay if I use this schema for
> reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> the result is :
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "children":["WMB1","WMB2"]
> },
> {
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 4.But if i want to igonre the "children" part instead of "device", I use
> this schema for reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> }
> ]
> }
> }
> }
> }
> ]
> }
> Unfortunately,I get exception:
> java.lang.ArrayIndexOutOfBoundsException: -8
> cause:java.lang.ArrayIndexOutOfBoundsException
> at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:122)
> at org.apache.avro.io.BinaryDecoder.skipString(BinaryDecoder.java:262)
> at org.apache.avro.io.ValidatingDecoder.skipString(ValidatingDecoder.java:113)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:60)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipRepeater(SkipParser.java:83)
> at org.apache.avro.io.ValidatingDecoder.skipArray(ValidatingDecoder.java:195)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:70)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipSymbol(SkipParser.java:93)
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:226)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at
> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:162)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:196)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:140)
> at
> org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:233)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141)
> at
> org.apache.avro.generic
[jira] [Assigned] (AVRO-793) A strange problem when I am trying to read avro record with a subset of the schema.
[
https://issues.apache.org/jira/browse/AVRO-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. reassigned AVRO-793:
Assignee: Thiruvalluvan M. G.
> A strange problem when I am trying to read avro record with a subset of the
> schema.
> ---
>
> Key: AVRO-793
> URL: https://issues.apache.org/jira/browse/AVRO-793
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.5.0
> Environment: Avro1.5,Windows xp/Ubuntu 10.0.4
>Reporter: Yingzhong Xu
>Assignee: Thiruvalluvan M. G.
>Priority: Critical
> Labels: Avro, Reading, Schema, Write
> Fix For: 1.5.1
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hi, all. When I am trying to read avro file with a subset of that
> schema(because I do not need all the details).I meet a strange problem.
> 1.I write data using this schema:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> },
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> 2.Here is a JSONObject for that schema.
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "device":"WMA1",
> "children":["WMB1","WMB2"]
> },
> {
> "device":"WMA2",
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 3.I write that record succefully.And it is okay if I use this schema for
> reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "children",
> "type": {
> "type": "array",
> "items": "string"
> }
> }
> ]
> }
> }
> }
> }
> ]
> }
> the result is :
> {
> "timestamp":1234567890,
> "type":{
> "WMA":[
> {
> "children":["WMB1","WMB2"]
> },
> {
> "children":["WMB1","WMB2"]
> }
> ]
> }
> }
> 4.But if i want to igonre the "children" part instead of "device", I use
> this schema for reading:
> {
> "name": "relation",
> "type": "record",
> "fields": [
> {
> "name": "timestamp",
> "type": "long"
> },
> {
> "name": "type",
> "type": {
> "type": "map",
> "values":{
> "type" : "array",
> "items": {
> "type":"record",
> "name":"sdf",
> "fields": [
> {
> "name": "device",
> "type": "string"
> }
> ]
> }
> }
> }
> }
> ]
> }
> Unfortunately,I get exception:
> java.lang.ArrayIndexOutOfBoundsException: -8
> cause:java.lang.ArrayIndexOutOfBoundsException
> at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:122)
> at org.apache.avro.io.BinaryDecoder.skipString(BinaryDecoder.java:262)
> at org.apache.avro.io.ValidatingDecoder.skipString(ValidatingDecoder.java:113)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:60)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipRepeater(SkipParser.java:83)
> at org.apache.avro.io.ValidatingDecoder.skipArray(ValidatingDecoder.java:195)
> at org.apache.avro.io.ParsingDecoder.skipTopSymbol(ParsingDecoder.java:70)
> at org.apache.avro.io.parsing.SkipParser.skipTo(SkipParser.java:71)
> at org.apache.avro.io.parsing.SkipParser.skipSymbol(SkipParser.java:93)
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:226)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at
> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:162)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:196)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:140)
> at
> org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:233)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:167)
>
[jira] [Commented] (AVRO-806) add a column-major codec for data files
[
https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022593#comment-13022593
]
Thiruvalluvan M. G. commented on AVRO-806:
--
It will be nice if we can make the column datafile compatible with the current
datafile standard (with the column's schema, of course). There will be a "meta"
datafile as a catalog of its constituent column data files. A single datafile
is, by definition, its own catalog. This approach, which I suppose is not
incompatible with the original proposal, has some advantages:
- We can reuse almost all the current code.
- Whether to split columns into further columns is completely left the
writer as both are valid.
However, this approach will make splitting unions harder. We have to cater to
"holes" in the branch columns. With a little bit of space overhead, we can
address the problem: [A, B, C] will be split into [null, A], [null, B] and
[null, C]. (A, B, or C is null, there won't be a corresponding column
datafile). The "holes" in the column will then be encoded as null branch. We do
not need a separate branch-index column.
An array of record { A, B, C } can be decomposed as three columns: array of A,
array of B and and array C. The idea can be extended to maps.
The good thing about this is that this defines a recursive column splitting
algorithm. And it is flexible. For example, if for a record {A, B, C} B and C
always come together, we can split it as A and {B, C}.
On another point, while it may be a good strategy for the writer to write
column blocks of data in lock-step fashion, the reader should not rely on it.
This allows different columns to be written at different points in time. This
will perform better because different columns have different sizes and compress
differently. But with this, skipping to next sync on one column could lead to
skipping to the middle of another column. That could be challenging.
> add a column-major codec for data files
> ---
>
> Key: AVRO-806
> URL: https://issues.apache.org/jira/browse/AVRO-806
> Project: Avro
> Issue Type: New Feature
> Components: java, spec
>Reporter: Doug Cutting
>
> Define a codec that, when a data file's schema is a record schema, writes
> blocks within the file in column-major order. This would permit better
> compression and also permit efficient skipping of fields that are not of
> interest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017884#comment-13017884 ] Thiruvalluvan M. G. commented on AVRO-792: -- I'll wait for the confirmation that this indeed solves the problem before I commit. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792-2.patch, AVRO-792-3.patch, AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumRea
[jira] [Updated] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-792: - Attachment: AVRO-792-3.patch I think the simplest solution is to move applyAliases invocation to ResolvingDecoder. Please see AVRO-792-3.patch. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792-2.patch, AVRO-792-3.patch, AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(Generi
[jira] [Commented] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016864#comment-13016864 ] Thiruvalluvan M. G. commented on AVRO-792: -- bq. Lastly, this contains an API change, making a public method private. This is probably a good thing here – the method in question was one I did not know what to do with previously and was a bit opaque for public consumption. This might not be acceptable for 1.5.1 and need to remain public though we can mark it deprecated and plan to hide it for 1.6.0. The only use of the return value of ResolvingDecoder.resolve() is to pass it the constructor of ResolvingDecoder, which is private. In theory someone could have used the resolve() method, but there is no practical value for that method outside ResolvingDecoder. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792-2.patch, AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecode
[jira] [Updated] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-792: - Attachment: AVRO-792-2.patch With this patch I don't see any significant performance difference. Scott, can you please verify. Thanks > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792-2.patch, AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) >
[jira] [Updated] (AVRO-795) C++ Datafile reader makes it hard to build adaptive clients
[ https://issues.apache.org/jira/browse/AVRO-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-795: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1089131. Thanks Doug for reviewing it. > C++ Datafile reader makes it hard to build adaptive clients > --- > > Key: AVRO-795 > URL: https://issues.apache.org/jira/browse/AVRO-795 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-795.patch > > > The current design of data file requires the C++ type for the data to be read > is passed as a template argument to the DataFileReader class. > This makes it hard for the client who want to examine the schema within the > data file before deciding the C++ data type. The forthcoming patch addresses > this problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-794) Makefile.am is no longer required in C++
[ https://issues.apache.org/jira/browse/AVRO-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-794: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1089128. Thanks Doug for reviewing it. > Makefile.am is no longer required in C++ > > > Key: AVRO-794 > URL: https://issues.apache.org/jira/browse/AVRO-794 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. >Priority: Minor > Fix For: 1.5.1 > > Attachments: AVRO-794.sh > > > Since we have moved to cmake, Makefile.am (which was used with automake) is > no longer required. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015941#comment-13015941 ] Thiruvalluvan M. G. commented on AVRO-792: -- I'll look into the performance issue. In the mean time, Ey-Chih Chow, can you check if this patch solves your problem? Thanks. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericD
[jira] [Updated] (AVRO-795) C++ Datafile reader makes it hard to build adaptive clients
[ https://issues.apache.org/jira/browse/AVRO-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-795: - Attachment: AVRO-795.patch This patch addresses the problem by replacing inheritance with composition. > C++ Datafile reader makes it hard to build adaptive clients > --- > > Key: AVRO-795 > URL: https://issues.apache.org/jira/browse/AVRO-795 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-795.patch > > > The current design of data file requires the C++ type for the data to be read > is passed as a template argument to the DataFileReader class. > This makes it hard for the client who want to examine the schema within the > data file before deciding the C++ data type. The forthcoming patch addresses > this problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-795) C++ Datafile reader makes it hard to build adaptive clients
[ https://issues.apache.org/jira/browse/AVRO-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-795: - Status: Patch Available (was: Open) > C++ Datafile reader makes it hard to build adaptive clients > --- > > Key: AVRO-795 > URL: https://issues.apache.org/jira/browse/AVRO-795 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-795.patch > > > The current design of data file requires the C++ type for the data to be read > is passed as a template argument to the DataFileReader class. > This makes it hard for the client who want to examine the schema within the > data file before deciding the C++ data type. The forthcoming patch addresses > this problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-795) C++ Datafile reader makes it hard to build adaptive clients
C++ Datafile reader makes it hard to build adaptive clients --- Key: AVRO-795 URL: https://issues.apache.org/jira/browse/AVRO-795 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.5.1 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Fix For: 1.5.1 The current design of data file requires the C++ type for the data to be read is passed as a template argument to the DataFileReader class. This makes it hard for the client who want to examine the schema within the data file before deciding the C++ data type. The forthcoming patch addresses this problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-794) Makefile.am is no longer required in C++
[ https://issues.apache.org/jira/browse/AVRO-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-794: - Attachment: AVRO-794.sh The attached shell script removes Makefile.am. > Makefile.am is no longer required in C++ > > > Key: AVRO-794 > URL: https://issues.apache.org/jira/browse/AVRO-794 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. >Priority: Minor > Fix For: 1.5.1 > > Attachments: AVRO-794.sh > > > Since we have moved to cmake, Makefile.am (which was used with automake) is > no longer required. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-794) Makefile.am is no longer required in C++
[ https://issues.apache.org/jira/browse/AVRO-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-794: - Status: Patch Available (was: Open) > Makefile.am is no longer required in C++ > > > Key: AVRO-794 > URL: https://issues.apache.org/jira/browse/AVRO-794 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. >Priority: Minor > Fix For: 1.5.1 > > > Since we have moved to cmake, Makefile.am (which was used with automake) is > no longer required. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-794) Makefile.am is no longer required in C++
Makefile.am is no longer required in C++ Key: AVRO-794 URL: https://issues.apache.org/jira/browse/AVRO-794 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.5.0 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Priority: Minor Fix For: 1.5.1 Since we have moved to cmake, Makefile.am (which was used with automake) is no longer required. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-792: - Attachment: AVRO-792.patch This patch fixes the problem. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Attachments: AVRO-792.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException: 40 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSeriali
[jira] [Updated] (AVRO-792) map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-792: - Fix Version/s: 1.5.1 Assignee: Thiruvalluvan M. G. Status: Patch Available (was: Open) The problem is because we are reusing ResolvingDecoder objects. ResolvingDecoder objects are stateful and hence cannot be used simultaneously to decode more than one stream. But constructing ResolvingDecoder is expensive. So we cannot construct ResolvingDecoder objects whenever we need. The real expensive part of ResolvingDecoder construction is construction of the symbol table (actually a symbol tree). The symbol tree is stateless (in fact it is immutable). Once the symbol tree is ready, construction of ResolvingDecoder is cheap. Before the DecoderFactory refacroring, GenericDatumReader used to cache the symbol tree (which was exposed through ResolvingDecoder.resolve() method.) The forthcoming patch fixes the problem by caching the symbol tree within the ResolvingDecoder. > map reduce job for avro 1.5 generates ArrayIndexOutOfBoundsException > > > Key: AVRO-792 > URL: https://issues.apache.org/jira/browse/AVRO-792 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: Mac with VMWare running Linux training-vm-Ubuntu >Reporter: ey-chih chow >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.1 > > Original Estimate: 504h > Remaining Estimate: 504h > > We have an avro map/reduce job used to be working with avro 1.4, but broken > with avro 1.5. The M/R job with avro 1.5 worked fine under our debugging > environment, but broken when we moved to a real cluster. At one instance f > testing, the job had 23 reducers. Four of them succeeded and the rest failed > because of the ArrayIndexOutOfBoundsException generated. Here are two > instances of the stack traces: > = > java.lang.ArrayIndexOutOfBoundsException: -1576799025 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86) > at > org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242) > at > org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) > at > com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) > at > org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) > at org.apache.hadoop.mapred.Child$4.run(Child.java:240) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:234) > = > java.lang.ArrayIndexOutOfBoundsException:
[jira] [Updated] (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-533: - Attachment: AVRO-533_3.patch Microsoft Visual Studio thinks the solution has just two projects not three. It does not recognize Avro.main project. The attached patch AVRO-533_3.patch fixes it. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Dona Alvarez > Fix For: 1.5.1 > > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > AVRO-533_3.patch, csharp.zip, csharp_2nd_patch.zip, csharp_3rd.zip, > csharp_4.zip, dotnet.patch, failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014097#comment-13014097 ] Thiruvalluvan M. G. commented on AVRO-533: -- I tested the .net version for compatibility with Java using different data types. I also tested its ability to read Java's blocking binary encoding. Both the tests pass. +1 for committing this. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Dona Alvarez > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > csharp.zip, csharp_2nd_patch.zip, csharp_3rd.zip, csharp_4.zip, dotnet.patch, > failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013791#comment-13013791 ] Thiruvalluvan M. G. commented on AVRO-533: -- I don't think anyone has done interop tests with other language implementations. I'm sure most of it would pass; the only thing that might cause trouble is encoding of double and float. If there is indeed incompatibility that would require data migration for the .net users in the future. I'll try to get it done today. I suggest that this Jira is assigned to Dona before committing this. I don't know how to do that, since the assignees list does not have her name. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > csharp.zip, csharp_2nd_patch.zip, csharp_3rd.zip, csharp_4.zip, dotnet.patch, > failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-789) Datafile support in C++
[ https://issues.apache.org/jira/browse/AVRO-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-789. -- Resolution: Fixed Committed revision 1086866 after adding a couple of more tests and ensuring that the data file is compatible to that used by Avro Java. Thank you Doug for reviewing. > Datafile support in C++ > --- > > Key: AVRO-789 > URL: https://issues.apache.org/jira/browse/AVRO-789 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.6.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-789.patch > > > The forthcoming patch adds Datafile support in C++. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-781) Generic data support in C++
[ https://issues.apache.org/jira/browse/AVRO-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012842#comment-13012842 ] Thiruvalluvan M. G. commented on AVRO-781: -- +1, yes I think we should include this in 1.5.1. > Generic data support in C++ > --- > > Key: AVRO-781 > URL: https://issues.apache.org/jira/browse/AVRO-781 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-781.patch > > > C++ does not currently have support for generic objects the way Java has. The > forthcoming patch adds that support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-783) Specifc object support in C++
[ https://issues.apache.org/jira/browse/AVRO-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012839#comment-13012839 ] Thiruvalluvan M. G. commented on AVRO-783: -- +1. One major feature that is missing in Avro C++ is Avro RPC. But I don't think we can add that in the next release. > Specifc object support in C++ > - > > Key: AVRO-783 > URL: https://issues.apache.org/jira/browse/AVRO-783 > Project: Avro > Issue Type: Improvement >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-783.patch > > > Forthcoming patch provides a way to encode and decode user objects and a code > generator that generates classes for given Avro schema. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-789) Datafile support in C++
[ https://issues.apache.org/jira/browse/AVRO-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012840#comment-13012840 ] Thiruvalluvan M. G. commented on AVRO-789: -- +1 for inclusion in 1.5.1 > Datafile support in C++ > --- > > Key: AVRO-789 > URL: https://issues.apache.org/jira/browse/AVRO-789 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.6.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-789.patch > > > The forthcoming patch adds Datafile support in C++. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012838#comment-13012838 ] Thiruvalluvan M. G. commented on AVRO-533: -- bq. Any idea what I've done wrong? Your version of NUnit is old. The Avro unit tests use NUnit 2.5 features extensively. Since NUnit tests use C# attributes (roughly equivalent to Java annotations), the old NUnit runner does not recognise the attributes of the new version; it doesn't fail either. It'll be nice if we can somehow make the tests report error if run with older runner. My quick search didn't reveal any easy solution. NUnit 2.5 or later should be able to run the tests. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > csharp.zip, csharp_2nd_patch.zip, csharp_3rd.zip, csharp_4.zip, dotnet.patch, > failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-789) Datafile support in C++
[ https://issues.apache.org/jira/browse/AVRO-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-789: - Component/s: c++ Affects Version/s: 1.6.0 > Datafile support in C++ > --- > > Key: AVRO-789 > URL: https://issues.apache.org/jira/browse/AVRO-789 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.6.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-789.patch > > > The forthcoming patch adds Datafile support in C++. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-789) Datafile support in C++
[ https://issues.apache.org/jira/browse/AVRO-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-789: - Attachment: AVRO-789.patch The patch adds Datafile support for C++. Tested on: - Cygwin - Ubuntu 10.10 - RHEL 4.8 > Datafile support in C++ > --- > > Key: AVRO-789 > URL: https://issues.apache.org/jira/browse/AVRO-789 > Project: Avro > Issue Type: Improvement >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-789.patch > > > The forthcoming patch adds Datafile support in C++. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-789) Datafile support in C++
Datafile support in C++ --- Key: AVRO-789 URL: https://issues.apache.org/jira/browse/AVRO-789 Project: Avro Issue Type: Improvement Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. The forthcoming patch adds Datafile support in C++. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-533: - Attachment: csharp_4.zip This is same as Dona's csharp_3rd.zip with the following changes: - The code now compiles and tests pass under Mono in Ubuntu 10.10 as well as Windows XP/Visual C# Express 2010. - The .csproj files were referring to TargetFrameworkVersion 4.0, which does not work with Mono. Blindly replacing it with 3.5 fails to work in Windows. I made TargetFrameworkVersion configurable. Please see README. - The code was using Enum.TryParse() which is not available in 3.5. I replaced it with Enum.Parse() - The Newtonsoft.Json.dll was not the latest. It was causing some errors. - Some of the DLL references in csproj were incorrect. I suppose it worked in Windows because visual studio was pulling them using registry. - Code Generation and Specific tests used Microsoft.CSharp.dll. I made them use either Mono.CSharp.dll or Microsoft.CSharp.dll depending on the Runtime. - Made all text files use unix line endings. It was mixed earlier. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > csharp.zip, csharp_2nd_patch.zip, csharp_3rd.zip, csharp_4.zip, dotnet.patch, > failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-783) Specifc object support in C++
[ https://issues.apache.org/jira/browse/AVRO-783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-783: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1085921 after adding a few more tests. Tested on RHEL 4.8, Ubuntu 10.10 and Cygwin. Thanks Doug for reviewing. > Specifc object support in C++ > - > > Key: AVRO-783 > URL: https://issues.apache.org/jira/browse/AVRO-783 > Project: Avro > Issue Type: Improvement >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-783.patch > > > Forthcoming patch provides a way to encode and decode user objects and a code > generator that generates classes for given Avro schema. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008905#comment-13008905 ] Thiruvalluvan M. G. commented on AVRO-533: -- Thanks a lot Dona for taking this up. I didn't do a proper review of the design or implementation. But I did build and had a cursory look. Here are my observations: - I could cleanly compile on my Windows XP with Visual Studio Express 2010. - Two unit tests failed. (TestResolutionMismatch_simple of GenericTests and SpecificTests). They failed because lines 123 to 126 in GenericReader.cs were commented out. Uncommenting that fixes the problem. - The test for codegen has its assertions commented out. Uncommenting it still succeeds, I'm not sure why they are commented. But still, I get an impression that the tests there is inadequate. It merely tests that compilation of the generated code goes through. It doesn't seem to test the contents of generated code. - Perhaps I'm missing something completely. To me, SpecificTests seems to test Generic rather than Specific. What am I missing? - Protocol compiler is in place. But there is no server or client implementation. This is fine, but I just wanted to check if my understanding is correct. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > csharp.zip, csharp_2nd_patch.zip, dotnet.patch, failure.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-783) Specifc object support in C++
[ https://issues.apache.org/jira/browse/AVRO-783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-783: - Status: Patch Available (was: Open) > Specifc object support in C++ > - > > Key: AVRO-783 > URL: https://issues.apache.org/jira/browse/AVRO-783 > Project: Avro > Issue Type: Improvement >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-783.patch > > > Forthcoming patch provides a way to encode and decode user objects and a code > generator that generates classes for given Avro schema. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-783) Specifc object support in C++
[ https://issues.apache.org/jira/browse/AVRO-783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-783: - Attachment: AVRO-783.patch This patch support encoding and decoding specific objects and code generator for given Avro schema. > Specifc object support in C++ > - > > Key: AVRO-783 > URL: https://issues.apache.org/jira/browse/AVRO-783 > Project: Avro > Issue Type: Improvement >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-783.patch > > > Forthcoming patch provides a way to encode and decode user objects and a code > generator that generates classes for given Avro schema. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (AVRO-783) Specifc object support in C++
Specifc object support in C++ - Key: AVRO-783 URL: https://issues.apache.org/jira/browse/AVRO-783 Project: Avro Issue Type: Improvement Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Forthcoming patch provides a way to encode and decode user objects and a code generator that generates classes for given Avro schema. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-781) Generic data support in C++
[ https://issues.apache.org/jira/browse/AVRO-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-781: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1083246. Thanks Doug for testing it. > Generic data support in C++ > --- > > Key: AVRO-781 > URL: https://issues.apache.org/jira/browse/AVRO-781 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-781.patch > > > C++ does not currently have support for generic objects the way Java has. The > forthcoming patch adds that support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-781) Generic data support in C++
[ https://issues.apache.org/jira/browse/AVRO-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-781: - Attachment: AVRO-781.patch The patch offers generic datum suppport for C++. > Generic data support in C++ > --- > > Key: AVRO-781 > URL: https://issues.apache.org/jira/browse/AVRO-781 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-781.patch > > > C++ does not currently have support for generic objects the way Java has. The > forthcoming patch adds that support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-781) Generic data support in C++
[ https://issues.apache.org/jira/browse/AVRO-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-781: - Status: Patch Available (was: Open) > Generic data support in C++ > --- > > Key: AVRO-781 > URL: https://issues.apache.org/jira/browse/AVRO-781 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.1 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-781.patch > > > C++ does not currently have support for generic objects the way Java has. The > forthcoming patch adds that support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (AVRO-781) Generic data support in C++
Generic data support in C++ --- Key: AVRO-781 URL: https://issues.apache.org/jira/browse/AVRO-781 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.5.1 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. C++ does not currently have support for generic objects the way Java has. The forthcoming patch adds that support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-775) File handle leak in DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003711#comment-13003711 ] Thiruvalluvan M. G. commented on AVRO-775: -- +1. Doug's implementation is cleaner and more general. > File handle leak in DataFileReader > -- > > Key: AVRO-775 > URL: https://issues.apache.org/jira/browse/AVRO-775 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: all >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: AVRO-775.patch, AVRO-775.patch > > > When a DataFileReader is constructed by passing it a java.io.File and the is > subsequently closed, the underlying FileInputStream is not getting closed. It > manifests as an error in mapreduce tests in Windows machines. The files > produced by mappers/reducers are opened as DataFiles for checking and, due to > this bug, the handles are not closed. So subsequent delete operation on the > file fails. And future mapper/reducer complains that the output already > exists. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-775) File handle leak in DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-775: - Status: Patch Available (was: Open) > File handle leak in DataFileReader > -- > > Key: AVRO-775 > URL: https://issues.apache.org/jira/browse/AVRO-775 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: all >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-775.patch > > > When a DataFileReader is constructed by passing it a java.io.File and the is > subsequently closed, the underlying FileInputStream is not getting closed. It > manifests as an error in mapreduce tests in Windows machines. The files > produced by mappers/reducers are opened as DataFiles for checking and, due to > this bug, the handles are not closed. So subsequent delete operation on the > file fails. And future mapper/reducer complains that the output already > exists. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-775) File handle leak in DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-775: - Attachment: AVRO-775.patch This patch addresses the problem. > File handle leak in DataFileReader > -- > > Key: AVRO-775 > URL: https://issues.apache.org/jira/browse/AVRO-775 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.5.0 > Environment: all >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.1 > > Attachments: AVRO-775.patch > > > When a DataFileReader is constructed by passing it a java.io.File and the is > subsequently closed, the underlying FileInputStream is not getting closed. It > manifests as an error in mapreduce tests in Windows machines. The files > produced by mappers/reducers are opened as DataFiles for checking and, due to > this bug, the handles are not closed. So subsequent delete operation on the > file fails. And future mapper/reducer complains that the output already > exists. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (AVRO-775) File handle leak in DataFileReader
File handle leak in DataFileReader -- Key: AVRO-775 URL: https://issues.apache.org/jira/browse/AVRO-775 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.5.0 Environment: all Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Fix For: 1.5.1 When a DataFileReader is constructed by passing it a java.io.File and the is subsequently closed, the underlying FileInputStream is not getting closed. It manifests as an error in mapreduce tests in Windows machines. The files produced by mappers/reducers are opened as DataFiles for checking and, due to this bug, the handles are not closed. So subsequent delete operation on the file fails. And future mapper/reducer complains that the output already exists. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed revision 1072677 after adjusting fot AVRO-700. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Fix For: 1.5.0 > > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch, > AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Priority: Blocker (was: Major) > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. >Priority: Blocker > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch, > AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (AVRO-753) Java: Improve BinaryEncoder Performance
[ https://issues.apache.org/jira/browse/AVRO-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992830#comment-12992830 ] Thiruvalluvan M. G. edited comment on AVRO-753 at 2/10/11 1:44 AM: --- We just write (positive) 1 for item-count and no byte-count (line 602 in BlockedBinaryEncoder.java) for overflow blocks. Since the reader is not supposed to expect byte-count when it encounters a positive item-count, it works. was (Author: thiru_mg): We just write 1 (line 602 in BlockedBinaryEncoder) for overflow blocks. We use positive 1 to indicate that we have a single item in the block and byte-count is unknown. > Java: Improve BinaryEncoder Performance > > > Key: AVRO-753 > URL: https://issues.apache.org/jira/browse/AVRO-753 > Project: Avro > Issue Type: Improvement > Components: java >Reporter: Scott Carey >Assignee: Scott Carey > Fix For: 1.5.0 > > Attachments: AVRO-753.v1.patch > > > BinaryEncoder has not had a performance improvement pass like BinaryDecoder > did. It still mostly writes directly to the underlying OutputStream which is > not optimal for performance. I like to use a rule that if you are writing to > an OutputStream or reading from an InputStream in chunks smaller than 128 > bytes, you have a performance problem. > Measurements indicate that optimizing BinaryEncoder yields a 2.5x to 6x > performance improvement. The process is significantly simpler than > BinaryDecoder because 'pushing' is easier than 'pulling' -- and also because > we do not need a 'direct' variant because BinaryEncoder already buffers > sometimes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-753) Java: Improve BinaryEncoder Performance
[ https://issues.apache.org/jira/browse/AVRO-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992830#comment-12992830 ] Thiruvalluvan M. G. commented on AVRO-753: -- We just write 1 (line 602 in BlockedBinaryEncoder) for overflow blocks. We use positive 1 to indicate that we have a single item in the block and byte-count is unknown. > Java: Improve BinaryEncoder Performance > > > Key: AVRO-753 > URL: https://issues.apache.org/jira/browse/AVRO-753 > Project: Avro > Issue Type: Improvement > Components: java >Reporter: Scott Carey >Assignee: Scott Carey > Fix For: 1.5.0 > > Attachments: AVRO-753.v1.patch > > > BinaryEncoder has not had a performance improvement pass like BinaryDecoder > did. It still mostly writes directly to the underlying OutputStream which is > not optimal for performance. I like to use a rule that if you are writing to > an OutputStream or reading from an InputStream in chunks smaller than 128 > bytes, you have a performance problem. > Measurements indicate that optimizing BinaryEncoder yields a 2.5x to 6x > performance improvement. The process is significantly simpler than > BinaryDecoder because 'pushing' is easier than 'pulling' -- and also because > we do not need a 'direct' variant because BinaryEncoder already buffers > sometimes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-700. -- Resolution: Fixed Took care both the minor problems Doug noted and committed revision 1065352. > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, AVRO-700.patch, AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987905#action_12987905 ] Thiruvalluvan M. G. commented on AVRO-533: -- Hi Dona, The patch (AVRo-533.patch) here is derived from Jeremy's original work (AVRO-533.zip). I refactored a lot, fixed functionality gaps, got in quite a bit of unit testing (over 80% coverage when I last measured) and it has a cleaner design. Also it works with both Microsoft.NET and Mono. I suggest you start from AVRO-533.patch. If you are interested in basic Avro encoding and decoding, the code in this patch should be good enough except for one issue - when resolving schemas it does not insert default values (for fields that reader's schema has but missing in writer's schema). Unimplemented functionality: (1) Avro data file format and (2) Avro RPC. (1) should be easy to implement as the work is not much and current design lends itself for data file (2) requires much more work. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > dotnet.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: AVRO-700.patch Doug, I'm sorry it took so many cycles to get this trough. Hope this is the last. Thanks for your patience. CoecTests and StreamTests were leaked from AVRO-711. Removed them. Now that tgz file is created both under build and dist. > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, AVRO-700.patch, AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: AVRO-700.patch This patch addresses both the issues Doug reported. > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980233#action_12980233 ] Thiruvalluvan M. G. edited comment on AVRO-711 at 1/11/11 1:29 PM: --- This is functionally same as the previous patch. But it uses IO streams instead of buffers. This improves performance and paves the way to implement Avro container files. This patch has been tested on - Ubuntu 10.04 - RHEL 4.8 - Cygwin was (Author: thiru_mg): This is functionally same as the previous patch. But it uses IO streams instead of buffers. This improves performance and paves the way to implement Avro container files. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch, > AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: AVRO-711.patch This is functionally same as the previous patch. But it uses IO streams instead of buffers. This improves performance and paves the way to implement Avro container files. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch, > AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: AVRO-711.patch Slightly better than the previous one. There was a CMake-2.6-specific variation. Removed it. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: (was: AVRO-711.patch) > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: AVRO-711.patch Same as the previous patch. But this is tested on the following: - RHEL 4.7 with CMake 2.6, Boost 1.41.0 - Ubuntu 10.04 with CMake 2.8.0, Boost 1.38.0 - Cygwin 1.7 with CMake 2.8.2, Boost 1.43.0 > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch, AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: AVRO-711.patch > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch, AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: NOTICE.txt This new patch handles recursive schemas and resolution. The only issue pending is that it does not insert default values for the fields that reader's schema has but writer's schema lacks. ValidSchema class does not yet have support for default values. We'll address these two together. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: (was: NOTICE.txt) > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972828#action_12972828 ] Thiruvalluvan M. G. commented on AVRO-533: -- Hi Petar, I'm really sorry, I couldn't find time to complete this. I'm presently working on Avro C++ as my company needs it. I plan to come back to this in Jan. > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > dotnet.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-663) avro-tools-1.4.0.jar doesn't meet the maven2 layout standard, making it inaccessble to maven users
[ https://issues.apache.org/jira/browse/AVRO-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971937#action_12971937 ] Thiruvalluvan M. G. commented on AVRO-663: -- +1 Some projects call it fatjar. In fact there is a maven plugin for fatjar: http://code.google.com/p/maven-fatjar-plugin/ But I've not used it myself. > avro-tools-1.4.0.jar doesn't meet the maven2 layout standard, making it > inaccessble to maven users > -- > > Key: AVRO-663 > URL: https://issues.apache.org/jira/browse/AVRO-663 > Project: Avro > Issue Type: Bug > Components: build >Affects Versions: 1.4.0 >Reporter: Brian Fox > > in /org/apache/avro/avro/1.4.0 you have avro-tools-1.4.0.jar. This should > instead be avro-1.4.0-tools.jar (similar to sources/javadoc jar structure) if > this is really just an attached artifact. If this is supposed to be something > separate and avro-tools is the artifactId, then it should be in > /org/apache/avro/avro-tools/1.4.0/avro-tools-1.4.0.jar -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970896#action_12970896 ] Thiruvalluvan M. G. edited comment on AVRO-711 at 12/13/10 12:04 PM: - This patch implements the JSON encoder and decoder for C++. The design is very similar to that of Java implementation. The patch is incomplete in two respects: - The decoder does not resolve schema. The Java implementation extends validating decoder to support resolution. I'll submit that as a separate patch. - The decoder and encoder do not handle recursive schema. They throw when a recursive schema is encountered. Again, I'll address this separately. Instead of modifying the existing interface, I added two new interfaces Encoder and Decoder in the same lines as Java. So none of the existing applications will have trouble. The new implementation uses the buffers and validSchema which we already have. As a side benefit there are binary encoder/decoder and validating encoder/decoder using the new interface. The new implementation is rigorously tested using the same test cases as that of Java. This patch may have conflicts with AVRO-700. I'll take care of them once AVRO-700 is done. was (Author: thiru_mg): This patch implements the JSON encoder and decoder for C++. The design is very similar to that of Java implementation. The patch is incomplete in two respects: - The decoder does not resolve schema. The Java implementation extends validating decoder to support resolution. I'll submit that as a separate patch. - The decoder and encoder do not handle recursive schema. They throw when a recursive schema is encountered. Again, I'll address this separately. Instead of modifying the existing interface, I added two new interfaces Encoder and Decoder in the same lines as Java. So none of the existing applications will have trouble. The new implementation uses the buffers and validSchema which we already have. As a side benefit there are binary encoder/decoder and validating encoder/decoder using the new interface. The new implementation is rigorously tested using the same test cases as that of Java. This patch may have conflicts with Avro-700. I'll take care of them once Avro-700 is done. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Assignee: Thiruvalluvan M. G. Status: Patch Available (was: Open) > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-711) JSON encoder and decoder for C++
[ https://issues.apache.org/jira/browse/AVRO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-711: - Attachment: AVRO-711.patch This patch implements the JSON encoder and decoder for C++. The design is very similar to that of Java implementation. The patch is incomplete in two respects: - The decoder does not resolve schema. The Java implementation extends validating decoder to support resolution. I'll submit that as a separate patch. - The decoder and encoder do not handle recursive schema. They throw when a recursive schema is encountered. Again, I'll address this separately. Instead of modifying the existing interface, I added two new interfaces Encoder and Decoder in the same lines as Java. So none of the existing applications will have trouble. The new implementation uses the buffers and validSchema which we already have. As a side benefit there are binary encoder/decoder and validating encoder/decoder using the new interface. The new implementation is rigorously tested using the same test cases as that of Java. This patch may have conflicts with Avro-700. I'll take care of them once Avro-700 is done. > JSON encoder and decoder for C++ > > > Key: AVRO-711 > URL: https://issues.apache.org/jira/browse/AVRO-711 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 > Environment: Linux, Cygwin >Reporter: Thiruvalluvan M. G. > Attachments: AVRO-711.patch > > > Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-711) JSON encoder and decoder for C++
JSON encoder and decoder for C++ Key: AVRO-711 URL: https://issues.apache.org/jira/browse/AVRO-711 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.5.0 Environment: Linux, Cygwin Reporter: Thiruvalluvan M. G. Forthcoming patch implements JSON encoder and decoder for C++. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: AVRO-700.patch The test paths were wrong in build.sh > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: (was: AVRO-700.patch) > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to CMake
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Summary: Change C++ build system to CMake (was: Change C++ build system to C++) > Change C++ build system to CMake > > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-705) DirectBinaryDecoder not correctly re-initialized with reuse option
[
https://issues.apache.org/jira/browse/AVRO-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-705:
-
Status: Patch Available (was: Open)
> DirectBinaryDecoder not correctly re-initialized with reuse option
> --
>
> Key: AVRO-705
> URL: https://issues.apache.org/jira/browse/AVRO-705
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Gilles Gaillard
>Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-705.patch
>
>
> Calling {{DecoderFactory.createBinaryDecoder(InputStream in, BinaryDecoder
> reuse)}} with {{reuse}} being a non null {{DirectBinaryDecoder}}, this
> {{DirectBinaryDecoder}} is simply reinitialized by a call to
> {{DirectBinaryDecoder.init}} which reinitializes the internal {{InputStream}}.
> The problem is that the {{byteReader}} attribute of {{DirectBinaryDecoder}}
> is not reinitialized and still points to the previously assigned
> {{ByteBufferInputStream}}
> The following is a proposition of correction at rev 1041110:
> {noformat}
> private ByteReader byteReader;
> DirectBinaryDecoder(InputStream in) {
> super();
> init(in);
> }
> @Override
> public void init(InputStream in) {
> this.in = in;
> byteReader = (in instanceof ByteBufferInputStream)
> ? new ReuseByteReader((ByteBufferInputStream) in)
> : new ByteReader();
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-705) DirectBinaryDecoder not correctly re-initialized with reuse option
[
https://issues.apache.org/jira/browse/AVRO-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-705:
-
Attachment: AVRO-705.patch
Good catch!
Added a test that catches the bug and fixed the bug as well.
> DirectBinaryDecoder not correctly re-initialized with reuse option
> --
>
> Key: AVRO-705
> URL: https://issues.apache.org/jira/browse/AVRO-705
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Gilles Gaillard
>Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-705.patch
>
>
> Calling {{DecoderFactory.createBinaryDecoder(InputStream in, BinaryDecoder
> reuse)}} with {{reuse}} being a non null {{DirectBinaryDecoder}}, this
> {{DirectBinaryDecoder}} is simply reinitialized by a call to
> {{DirectBinaryDecoder.init}} which reinitializes the internal {{InputStream}}.
> The problem is that the {{byteReader}} attribute of {{DirectBinaryDecoder}}
> is not reinitialized and still points to the previously assigned
> {{ByteBufferInputStream}}
> The following is a proposition of correction at rev 1041110:
> {noformat}
> private ByteReader byteReader;
> DirectBinaryDecoder(InputStream in) {
> super();
> init(in);
> }
> @Override
> public void init(InputStream in) {
> this.in = in;
> byteReader = (in instanceof ByteBufferInputStream)
> ? new ReuseByteReader((ByteBufferInputStream) in)
> : new ByteReader();
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to C++
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: ignored_files apply-patch.sh AVRO-700.patch A much cleaner version all files produced during make are put into build directory leaving lang/c++ clean. Please download ignored_files, apply_patch.sh and AVRO-700.patch into the top level Avro source directory. From within that directory, issue: sh ./apply_patch.sh patch -p0 < AVRO-700.patch > Change C++ build system to C++ > -- > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, apply-patch.sh, AVRO-700.patch, > AVRO-700.patch, ignored_files > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-706) Java: Type promotion not succeeding for long -> float
[
https://issues.apache.org/jira/browse/AVRO-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-706:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed revision 1041903.
Thanks Doug for reviewing the fix.
> Java: Type promotion not succeeding for long -> float
> -
>
> Key: AVRO-706
> URL: https://issues.apache.org/jira/browse/AVRO-706
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Scott Carey
>Assignee: Thiruvalluvan M. G.
> Fix For: 1.5.0
>
> Attachments: AVRO-706.patch
>
>
> A bug reported on the dev email list. I have not confirmed it:
> {quote}
> Hi,
> Avro spec says that writer schema's long can be promoted to reader
> schema's float or double. In my test, I'm getting ErrorAction from
> ResolvingGrammarGenerator.generate() when writer schema is long and
> reader schema expects float. Looking at the source code, when reader
> and writer type are not the same, the switch(readerType) only handles
> Long, Double and Union, not Float. This is the reason I'm getting "Found
> Long, expecting Float" error.
> Best regards,
> Xiaolu
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-705) DirectBinaryDecoder not correctly re-initialized with reuse option
[
https://issues.apache.org/jira/browse/AVRO-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. reassigned AVRO-705:
Assignee: Thiruvalluvan M. G.
> DirectBinaryDecoder not correctly re-initialized with reuse option
> --
>
> Key: AVRO-705
> URL: https://issues.apache.org/jira/browse/AVRO-705
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Gilles Gaillard
>Assignee: Thiruvalluvan M. G.
>
> Calling {{DecoderFactory.createBinaryDecoder(InputStream in, BinaryDecoder
> reuse)}} with {{reuse}} being a non null {{DirectBinaryDecoder}}, this
> {{DirectBinaryDecoder}} is simply reinitialized by a call to
> {{DirectBinaryDecoder.init}} which reinitializes the internal {{InputStream}}.
> The problem is that the {{byteReader}} attribute of {{DirectBinaryDecoder}}
> is not reinitialized and still points to the previously assigned
> {{ByteBufferInputStream}}
> The following is a proposition of correction at rev 1041110:
> {noformat}
> private ByteReader byteReader;
> DirectBinaryDecoder(InputStream in) {
> super();
> init(in);
> }
> @Override
> public void init(InputStream in) {
> this.in = in;
> byteReader = (in instanceof ByteBufferInputStream)
> ? new ReuseByteReader((ByteBufferInputStream) in)
> : new ByteReader();
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-706) Java: Type promotion not succeeding for long -> float
[
https://issues.apache.org/jira/browse/AVRO-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-706:
-
Attachment: AVRO-706.patch
There were two bugs:
1. int and long were not getting promoted to float
2. double was getting "promoted" to long, which is not in the spec.
The attached patch fixes both the bugs. There were already tests for (1) but
were commented (I can't remember why). I un-commented them. There was a test
verify the wrong behavior (2). I removed it after ensuring that it indeed fails
after the patch.
> Java: Type promotion not succeeding for long -> float
> -
>
> Key: AVRO-706
> URL: https://issues.apache.org/jira/browse/AVRO-706
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Scott Carey
>Assignee: Thiruvalluvan M. G.
> Fix For: 1.5.0
>
> Attachments: AVRO-706.patch
>
>
> A bug reported on the dev email list. I have not confirmed it:
> {quote}
> Hi,
> Avro spec says that writer schema's long can be promoted to reader
> schema's float or double. In my test, I'm getting ErrorAction from
> ResolvingGrammarGenerator.generate() when writer schema is long and
> reader schema expects float. Looking at the source code, when reader
> and writer type are not the same, the switch(readerType) only handles
> Long, Double and Union, not Float. This is the reason I'm getting "Found
> Long, expecting Float" error.
> Best regards,
> Xiaolu
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-706) Java: Type promotion not succeeding for long -> float
[
https://issues.apache.org/jira/browse/AVRO-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-706:
-
Assignee: Thiruvalluvan M. G.
Status: Patch Available (was: Open)
Thanks Xiaolu for pointing this out and Scott for creating Jira.
> Java: Type promotion not succeeding for long -> float
> -
>
> Key: AVRO-706
> URL: https://issues.apache.org/jira/browse/AVRO-706
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.4.1
>Reporter: Scott Carey
>Assignee: Thiruvalluvan M. G.
> Fix For: 1.5.0
>
>
> A bug reported on the dev email list. I have not confirmed it:
> {quote}
> Hi,
> Avro spec says that writer schema's long can be promoted to reader
> schema's float or double. In my test, I'm getting ErrorAction from
> ResolvingGrammarGenerator.generate() when writer schema is long and
> reader schema expects float. Looking at the source code, when reader
> and writer type are not the same, the switch(readerType) only handles
> Long, Double and Union, not Float. This is the reason I'm getting "Found
> Long, expecting Float" error.
> Best regards,
> Xiaolu
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-700) Change C++ build system to C++
[ https://issues.apache.org/jira/browse/AVRO-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-700: - Attachment: apply-patch.sh AVRO-700.patch To apply the patch, from top level source directory "patch -p0 < AVRO-700.patch" and then "sh ./apply-patch.sh" > Change C++ build system to C++ > -- > > Key: AVRO-700 > URL: https://issues.apache.org/jira/browse/AVRO-700 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.5.0 >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.5.0 > > Attachments: apply-patch.sh, AVRO-700.patch > > > If we move from the current automake to CMake, build becomes portable across > multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but > was crashing. I've tested this on Ubuntu 10.04 and Cygwin. > There are problems making it work with Visual Studio 2008 Express, not > because of CMake, but because the current build procedure needs Flex, Bison > and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-700) Change C++ build system to C++
Change C++ build system to C++ -- Key: AVRO-700 URL: https://issues.apache.org/jira/browse/AVRO-700 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.5.0 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Fix For: 1.5.0 If we move from the current automake to CMake, build becomes portable across multiple platforms. Prior to this patch, Avro C++ was building on Cygwin but was crashing. I've tested this on Ubuntu 10.04 and Cygwin. There are problems making it work with Visual Studio 2008 Express, not because of CMake, but because the current build procedure needs Flex, Bison and python. Visual Studio seems to have trouble with these three. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-690) Schema resolution error message for union lacks field name
[ https://issues.apache.org/jira/browse/AVRO-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-690: Assignee: Thiruvalluvan M. G. > Schema resolution error message for union lacks field name > -- > > Key: AVRO-690 > URL: https://issues.apache.org/jira/browse/AVRO-690 > Project: Avro > Issue Type: Improvement >Reporter: Ron Bodkin >Assignee: Thiruvalluvan M. G. > > I had a field whose type changed from ["null", "int"] to ["null", "string"]. > When reading a file with the old type (an int) in it, the Avro reader gave > this error message: > org.apache.avro.AvroTypeException: Found "int", expecting ["null","string"] > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:212) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:187) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:125) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:149) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:121) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:112) > at org.apache.avro.file.DataFileStream.next(DataFileStream.java:198) > at > org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67) > at > org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34) > It would be helpful if the generate method had context about what record and > field name it was working on, so the message could be > org.apache.avro.AvroTypeException: Found "int", expecting ["null","string"] > in field "my_field_name" in record "record_name" > I think this would mean adding a field and a record parameter to the > ResolvingGrammarGenerator.generate and resolve* methods so an error could > identify where it came from. > E.g., the data about the field was available in the resolveRecords methods in > this trace of where that error got generated. > Symbol.error(String) line: 122 > ResolvingGrammarGenerator.generate(Schema, Schema, Map) line: > 170 > ResolvingGrammarGenerator.resolveUnion(Schema, Schema, Map) > line: 186 > ResolvingGrammarGenerator.generate(Schema, Schema, Map) line: > 120 > ResolvingGrammarGenerator.resolveRecords(Schema, Schema, Map) > line: 254 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-533: - Attachment: AVRO-533.patch In terms of functionality, this patch is essentially the same as the previous one, but works on both Windows and Linux. The batch contains the following three libraries: Newtonsoft.Json.Net20.dll from http://james.newtonking.com/projects/json-net.aspx (Creative Commons Attribution 2.5 License) log4net.dll from http://logging.apache.org/log4net/ (Apache 2.0 license) nunit.framework.dll from http://www.nunit.org/ (zlib/libpng license) I've copied these files from a larger download. Can someone confirm that it's legal to copy the individual files and to use them in our project? Thanks > .NET implementation of Avro > --- > > Key: AVRO-533 > URL: https://issues.apache.org/jira/browse/AVRO-533 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Jeff Hammerbacher >Assignee: Thiruvalluvan M. G. > Attachments: AVRO-533.patch, AVRO-533.patch, AVRO-533.zip, > dotnet.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AVRO-664) Ruby server takes a long time to start in interop tests
[ https://issues.apache.org/jira/browse/AVRO-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909220#action_12909220 ] Thiruvalluvan M. G. edited comment on AVRO-664 at 9/14/10 11:31 AM: Thanks Doug for reviewing the patch. Increasing from 1 second time-out previously to 60 seconds in the patch appears too excessive. As Doug pointed out some serious problems may go unnoticed. I set the limit to 10 seconds in steps of 1-second and then committed revision 996877. was (Author: thiru_mg): Committed revision 996877. > Ruby server takes a long time to start in interop tests > --- > > Key: AVRO-664 > URL: https://issues.apache.org/jira/browse/AVRO-664 > Project: Avro > Issue Type: Bug > Components: ruby >Affects Versions: 1.4.0 > Environment: Ubuntu Lucid Lynx >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.4.1 > > Attachments: AVRO-664.patch > > > The ruby server uses a wrong key in the config (:Host instead of > :BindAddress) which cause Socket::getaddrinfo() take up to 15 seconds. The > shell script which starts the server waits for jist 1 second and test fails > abruptly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AVRO-664) Ruby server takes a long time to start in interop tests
[ https://issues.apache.org/jira/browse/AVRO-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-664. -- Resolution: Fixed Committed revision 996877. > Ruby server takes a long time to start in interop tests > --- > > Key: AVRO-664 > URL: https://issues.apache.org/jira/browse/AVRO-664 > Project: Avro > Issue Type: Bug > Components: ruby >Affects Versions: 1.4.0 > Environment: Ubuntu Lucid Lynx >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.4.1 > > Attachments: AVRO-664.patch > > > The ruby server uses a wrong key in the config (:Host instead of > :BindAddress) which cause Socket::getaddrinfo() take up to 15 seconds. The > shell script which starts the server waits for jist 1 second and test fails > abruptly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-664) Ruby server takes a long time to start in interop tests
[ https://issues.apache.org/jira/browse/AVRO-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-664: - Attachment: AVRO-664.patch The patch addresses both the problems. It fixes the name in tools.rb and makes the shell script wait for variable amount of time (up to 60 seconds). Either of the fixes is actually sufficient. But I thought an extra level of protection will not hurt. > Ruby server takes a long time to start in interop tests > --- > > Key: AVRO-664 > URL: https://issues.apache.org/jira/browse/AVRO-664 > Project: Avro > Issue Type: Bug > Components: ruby >Affects Versions: 1.4.0 > Environment: Ubuntu Lucid Lynx >Reporter: Thiruvalluvan M. G. >Assignee: Thiruvalluvan M. G. > Fix For: 1.4.1 > > Attachments: AVRO-664.patch > > > The ruby server uses a wrong key in the config (:Host instead of > :BindAddress) which cause Socket::getaddrinfo() take up to 15 seconds. The > shell script which starts the server waits for jist 1 second and test fails > abruptly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-664) Ruby server takes a long time to start in interop tests
Ruby server takes a long time to start in interop tests --- Key: AVRO-664 URL: https://issues.apache.org/jira/browse/AVRO-664 Project: Avro Issue Type: Bug Components: ruby Affects Versions: 1.4.0 Environment: Ubuntu Lucid Lynx Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Fix For: 1.4.1 The ruby server uses a wrong key in the config (:Host instead of :BindAddress) which cause Socket::getaddrinfo() take up to 15 seconds. The shell script which starts the server waits for jist 1 second and test fails abruptly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
