[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[
https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636484#comment-16636484
]
Raymie Stata commented on AVRO-2235:
I looked at the comment at the top of TestSpecificLogicalTypes and realized
that maybe I shouldn't have updated TestRecordWithLogicalTypes. That comment
reads:
{quote}The classes [TestRecordWithLogicalTypes and
TestRecordWithoutLogicalTypes]should not be re-generated because they test
compatibility of Avro with existing Avro-generated sources. When using classes
generated before AVRO-1684, logical types should not be applied by the read or
write paths. Those files should behave as they did before. At the same time,
[~nkollar] suggested in my (GitHub) pull request for AVRO-2090 that I
regenerate TestRecordForLogicalTypes.
{quote}
So it sounds like this code here is for testing backward compatibility and thus
shouldn't be updated.
At the same time, in my (GitHub) pull request for AVRO-2090, [~nkollar]
suggests that I _do_ regenerate these classes. At this point, I'm not sure
what is the right thing to do. Any suggestions?
> Regenerate TestRecordWithLogicalTypes
> -
>
> Key: AVRO-2235
> URL: https://issues.apache.org/jira/browse/AVRO-2235
> Project: Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Raymie Stata
>Priority: Major
>
> TestRecordWithLogicalTypes.java is code that was generated by the specific
> compiler and then moved into the testing code tree. It hasn't been changed
> in a while, although the compiler is evolving. I tried to regenerate it and
> found there is a problem with record_with_logical_types.avsc. I will fix the
> schema file and then regenerate TestRecordWithLogicalTypes and check both in.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636466#comment-16636466 ] Raymie Stata commented on AVRO-2235: I want to re-regenerate TestRecordForLogicalTypes.java with my modifications to the specific compiler, but would prefer to do that after successfully re-generating it on the master. > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636464#comment-16636464 ] Raymie Stata commented on AVRO-2235: Does anyone know how TestRecordWithoutLogicalTypes.java was generated? Was it a hand-edit of the generated code for TestRecordWithLogicalTypes.java? > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636450#comment-16636450 ] ASF GitHub Bot commented on AVRO-2235: -- rstata opened a new pull request #342: AVRO-2235: regenerate TestRecordWithLogicalTypes.java URL: https://github.com/apache/avro/pull/342 My original fix conflicted with patch for AVRO-2079. The AVRO-2079 fixed a typo in record_with_logical_types.avsc. However, while AVRO-2079 made a cosmetic change to RecordWithLogicalTypes.java, it did not regenerate it. I fixed another problem in record_with_logical_types.avsc and regenerated RecordWithLogicalTypes.java. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[
https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636418#comment-16636418
]
ASF GitHub Bot commented on AVRO-2235:
--
rstata closed pull request #341: Fix AVRO-2235
URL: https://github.com/apache/avro/pull/341
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/lang/java/avro/src/test/java/org/apache/avro/specific/TestRecordWithLogicalTypes.java
b/lang/java/avro/src/test/java/org/apache/avro/specific/TestRecordWithLogicalTypes.java
index 493a8e67c..851e49a2d 100644
---
a/lang/java/avro/src/test/java/org/apache/avro/specific/TestRecordWithLogicalTypes.java
+++
b/lang/java/avro/src/test/java/org/apache/avro/specific/TestRecordWithLogicalTypes.java
@@ -5,16 +5,16 @@
*/
package org.apache.avro.specific;
-import org.apache.avro.message.BinaryMessageDecoder;
+import org.apache.avro.specific.SpecificData;
import org.apache.avro.message.BinaryMessageEncoder;
+import org.apache.avro.message.BinaryMessageDecoder;
+import org.apache.avro.message.SchemaStore;
-import java.math.BigDecimal;
-
-@SuppressWarnings("all")
+/** Schema for TestRecordWithLogicalTypes and TestRecordWithoutLogicalTypes,
see TestSpecificLogicalTypes */
@org.apache.avro.specific.AvroGenerated
public class TestRecordWithLogicalTypes extends
org.apache.avro.specific.SpecificRecordBase implements
org.apache.avro.specific.SpecificRecord {
- private static final long serialVersionUID = -4211233492739285532L;
- public static final org.apache.avro.Schema SCHEMA$ = new
org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"TestRecordWithLogicalTypes\",\"namespace\":\"org.apache.avro.specific\",\"fields\":[{\"name\":\"b\",\"type\":\"boolean\"},{\"name\":\"i32\",\"type\":\"int\"},{\"name\":\"i64\",\"type\":\"long\"},{\"name\":\"f32\",\"type\":\"float\"},{\"name\":\"f64\",\"type\":\"double\"},{\"name\":\"s\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"d\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"t\",\"type\":{\"type\":\"int\",\"logicalType\":\"time-millis\"}},{\"name\":\"ts\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"dec\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":9,\"scale\":2}}]}");
+ private static final long serialVersionUID = 6236661767120588912L;
+ public static final org.apache.avro.Schema SCHEMA$ = new
org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"TestRecordWithLogicalTypes\",\"namespace\":\"org.apache.avro.specific\",\"doc\":\"Schema
for TestRecordWithLogicalTypes and TestRecordWithoutLogicalTypes, see
TestSpecificLogicalTypes\",\"fields\":[{\"name\":\"b\",\"type\":\"boolean\"},{\"name\":\"i32\",\"type\":\"int\"},{\"name\":\"i64\",\"type\":\"long\"},{\"name\":\"f32\",\"type\":\"float\"},{\"name\":\"f64\",\"type\":\"double\"},{\"name\":\"s\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"d\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"t\",\"type\":{\"type\":\"int\",\"logicalType\":\"time-millis\"}},{\"name\":\"ts\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"dec\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":9,\"scale\":2}}]}");
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
private static SpecificData MODEL$ = new SpecificData();
@@ -25,12 +25,38 @@
private static final BinaryMessageDecoder
DECODER =
new BinaryMessageDecoder(MODEL$, SCHEMA$);
- /** Serializes this ${schema.getName()} to a ByteBuffer. */
+ /**
+ * Return the BinaryMessageDecoder instance used by this class.
+ * @return the message decoder used by this class
+ */
+ public static BinaryMessageDecoder getDecoder() {
+return DECODER;
+ }
+
+ /**
+ * Create a new BinaryMessageDecoder instance for this class that uses the
specified {@link SchemaStore}.
+ * @param resolver a {@link SchemaStore} used to find schemas by fingerprint
+ * @return a BinaryMessageDecoder instance for this class backed by the
given SchemaStore
+ */
+ public static BinaryMessageDecoder
createDecoder(SchemaStore resolver) {
+return new BinaryMessageDecoder(MODEL$,
SCHEMA$, resolver);
+ }
+
+ /**
+ * Serializes this TestRecordWithLogicalTypes to a ByteBuffer.
+ * @return a buffer holding the serialized data for this instance
+ * @throws java.io.IOException if this instance could not be serialized
+ */
public java.nio.ByteBuffer toByteBuffer() throws java.io.IOException {
return ENCODER.encode(this);
}
- /** Deserializes a ${schema.getName()} from a ByteBuffer. */
+ /**
+ * Deserializes a
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636404#comment-16636404 ] ASF GitHub Bot commented on AVRO-2235: -- rstata opened a new pull request #341: Fix AVRO-2235 URL: https://github.com/apache/avro/pull/341 Fixed typo in record_with_logical_types.asvc, then regenerated TestRecordWithLogicalTypes.java This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
Raymie Stata created AVRO-2235: -- Summary: Regenerate TestRecordWithLogicalTypes Key: AVRO-2235 URL: https://issues.apache.org/jira/browse/AVRO-2235 Project: Avro Issue Type: Bug Components: logical types Reporter: Raymie Stata Assignee: Raymie Stata TestRecordWithLogicalTypes.java is code that was generated by the specific compiler and then moved into the testing code tree. It hasn't been changed in a while, although the compiler is evolving. I tried to regenerate it and found there is a problem with record_with_logical_types.avsc. I will fix the schema file and then regenerate TestRecordWithLogicalTypes and check both in. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2202) Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when combined with musl libc static builds
[ https://issues.apache.org/jira/browse/AVRO-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-2202: -- Status: Open (was: Patch Available) > Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when > combined with musl libc static builds > > > Key: AVRO-2202 > URL: https://issues.apache.org/jira/browse/AVRO-2202 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 > Environment: OS: Gentoo Linux > Compiler: Gcc 6.4.0 targeting musl libc 1.19 with stdlibc++ in a static build >Reporter: Josh Scoggins >Priority: Major > > Observed with master and version 1.8.2. Inside of the C++ implementation's > CMakeLists.txt the lines 56-60 were added sometime in 2016 which activate > features inside of libstdc++ to help find errors. This change has two side > effects: > # -g is eliminated from the build so debugging gdb becomes harder as source > position data has been removed > # The safety containers that implicitly wrap all standard containers somehow > (I can't figure out exactly how) cause memory corruption when setMetadata is > called (in version 1.8.2) or when a Writer object is destroyed (on master). > Since our software uses musl libc with libstdc++ I believe that it is an > interaction with these debugging features which cause memory corruption (as I > said, I have no clue _why_ it is happening, only that these features _cause_ > it :() > Previously, we were using 1.7.7 which did not have these lines in its > CMakeLists.txt and thus did not exhibit any sort of memory corruption. I was > able to work around this by removing lines 56-60 from CMakeLists.txt so our > static libs could be built and our unit tests not crash by writing to memory > it did not own. There should be a toggle or something similar to turn off > these features for debug builds as they are for developers of avro only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1335) C++ should support field default values
[
https://issues.apache.org/jira/browse/AVRO-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635907#comment-16635907
]
ASF GitHub Bot commented on AVRO-1335:
--
thiru-apache commented on a change in pull request #241: AVRO-1335: Adds C++
support for default values in schema serializatio…
URL: https://github.com/apache/avro/pull/241#discussion_r222054059
##
File path: lang/c++/impl/NodeImpl.cc
##
@@ -201,14 +216,177 @@ NodeRecord::printJson(std::ostream , int depth) const
os << indent(++depth) << "\"name\": \"" << leafNameAttributes_.get(i)
<< "\",\n";
os << indent(depth) << "\"type\": ";
leafAttributes_.get(i)->printJson(os, depth);
+
+if (!defaultValues.empty()) {
+ if (!defaultValues[i].isUnion() &&
+ defaultValues[i].type() == AVRO_NULL) {
+// No "default" field.
+ } else {
+os << ",\n" << indent(depth) << "\"default\": ";
+leafAttributes_.get(i)->printDefaultToJson(defaultValues[i], os,
+ depth);
+ }
+}
os << '\n';
os << indent(--depth) << '}';
}
os << '\n' << indent(--depth) << "]\n";
os << indent(--depth) << '}';
}
-void
+void NodePrimitive::printDefaultToJson(const GenericDatum , std::ostream ,
+ int depth) const {
Review comment:
Shouldn't `depth` be used for indenting each line?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C++ should support field default values
> ---
>
> Key: AVRO-1335
> URL: https://issues.apache.org/jira/browse/AVRO-1335
> Project: Avro
> Issue Type: Improvement
> Components: c++
>Affects Versions: 1.7.4
>Reporter: Bin Guo
>Assignee: Victor Mota
>Priority: Major
> Attachments: AVRO-1335.patch
>
>
> We found that resolvingDecoder could not provide bidirectional compatibility
> between different version of schemas.
> Especially for records, for example:
> {code:title=First schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> {code:title=Second schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> },
> {
> "name": "Version2",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> Say, node A knows only the first schema and node B knows the second schema,
> and the second schema has more fields.
> Any data generated by node B can be resolved by first schema 'cause the
> additional field is marked as skipped.
> But data generated by node A can not be resolved by second schema and throws
> an exception *"Don't know how to handle excess fields for reader."*
> This is because data is resolved exactly according to the auto-generated
> codec_traits which trying to read the excess field.
> The problem is we just can not only ignore the excess field in record, since
> the data after the troublesome record also needs to be resolved.
> Actually this problem stucked us for a very long time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1335) C++ should support field default values
[
https://issues.apache.org/jira/browse/AVRO-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635905#comment-16635905
]
ASF GitHub Bot commented on AVRO-1335:
--
thiru-apache commented on a change in pull request #241: AVRO-1335: Adds C++
support for default values in schema serializatio…
URL: https://github.com/apache/avro/pull/241#discussion_r222053332
##
File path: lang/c++/impl/NodeImpl.cc
##
@@ -17,11 +17,42 @@
*/
+#include
#include "NodeImpl.hh"
+
namespace avro {
-SchemaResolution
+namespace {
+// Escape double quotes (") for serialization.
+std::string escape(std::string s) {
+ boost::replace_all(s, "\"", "\\\"");
Review comment:
Shouldn't we escape `\` as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C++ should support field default values
> ---
>
> Key: AVRO-1335
> URL: https://issues.apache.org/jira/browse/AVRO-1335
> Project: Avro
> Issue Type: Improvement
> Components: c++
>Affects Versions: 1.7.4
>Reporter: Bin Guo
>Assignee: Victor Mota
>Priority: Major
> Attachments: AVRO-1335.patch
>
>
> We found that resolvingDecoder could not provide bidirectional compatibility
> between different version of schemas.
> Especially for records, for example:
> {code:title=First schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> {code:title=Second schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> },
> {
> "name": "Version2",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> Say, node A knows only the first schema and node B knows the second schema,
> and the second schema has more fields.
> Any data generated by node B can be resolved by first schema 'cause the
> additional field is marked as skipped.
> But data generated by node A can not be resolved by second schema and throws
> an exception *"Don't know how to handle excess fields for reader."*
> This is because data is resolved exactly according to the auto-generated
> codec_traits which trying to read the excess field.
> The problem is we just can not only ignore the excess field in record, since
> the data after the troublesome record also needs to be resolved.
> Actually this problem stucked us for a very long time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1335) C++ should support field default values
[
https://issues.apache.org/jira/browse/AVRO-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635904#comment-16635904
]
ASF GitHub Bot commented on AVRO-1335:
--
thiru-apache commented on a change in pull request #241: AVRO-1335: Adds C++
support for default values in schema serializatio…
URL: https://github.com/apache/avro/pull/241#discussion_r222053541
##
File path: lang/c++/impl/NodeImpl.cc
##
@@ -17,11 +17,42 @@
*/
+#include
#include "NodeImpl.hh"
+
namespace avro {
-SchemaResolution
+namespace {
+// Escape double quotes (") for serialization.
+std::string escape(std::string s) {
+ boost::replace_all(s, "\"", "\\\"");
+ return s;
+}
+
+// Wrap an indentation in a struct for ostream operator<<
+struct indent {
+indent(int depth) :
+d(depth)
+{ }
+int d;
+};
+
+/// ostream operator for indent
+std::ostream& operator <<(std::ostream , indent x)
+{
+static const std::string spaces("");
+while(x.d--) {
Review comment:
Styling: space before `(` for `if`, `for` and `while`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C++ should support field default values
> ---
>
> Key: AVRO-1335
> URL: https://issues.apache.org/jira/browse/AVRO-1335
> Project: Avro
> Issue Type: Improvement
> Components: c++
>Affects Versions: 1.7.4
>Reporter: Bin Guo
>Assignee: Victor Mota
>Priority: Major
> Attachments: AVRO-1335.patch
>
>
> We found that resolvingDecoder could not provide bidirectional compatibility
> between different version of schemas.
> Especially for records, for example:
> {code:title=First schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> {code:title=Second schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> },
> {
> "name": "Version2",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> Say, node A knows only the first schema and node B knows the second schema,
> and the second schema has more fields.
> Any data generated by node B can be resolved by first schema 'cause the
> additional field is marked as skipped.
> But data generated by node A can not be resolved by second schema and throws
> an exception *"Don't know how to handle excess fields for reader."*
> This is because data is resolved exactly according to the auto-generated
> codec_traits which trying to read the excess field.
> The problem is we just can not only ignore the excess field in record, since
> the data after the troublesome record also needs to be resolved.
> Actually this problem stucked us for a very long time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1335) C++ should support field default values
[
https://issues.apache.org/jira/browse/AVRO-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635906#comment-16635906
]
ASF GitHub Bot commented on AVRO-1335:
--
thiru-apache commented on a change in pull request #241: AVRO-1335: Adds C++
support for default values in schema serializatio…
URL: https://github.com/apache/avro/pull/241#discussion_r222050819
##
File path: lang/c++/api/NodeImpl.hh
##
@@ -538,6 +557,16 @@ inline NodePtr resolveSymbol(const NodePtr )
return symNode->getNode();
}
+template< typename T >
+inline std::string intToHex( T i )
Review comment:
Please take care of styling: No space after `(` or before `)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C++ should support field default values
> ---
>
> Key: AVRO-1335
> URL: https://issues.apache.org/jira/browse/AVRO-1335
> Project: Avro
> Issue Type: Improvement
> Components: c++
>Affects Versions: 1.7.4
>Reporter: Bin Guo
>Assignee: Victor Mota
>Priority: Major
> Attachments: AVRO-1335.patch
>
>
> We found that resolvingDecoder could not provide bidirectional compatibility
> between different version of schemas.
> Especially for records, for example:
> {code:title=First schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> {code:title=Second schema}
> {
> "type": "record",
> "name": "TestRecord",
> "fields": [
> {
> "name": "MyData",
> "type": {
> "type": "record",
> "name": "SubData",
> "fields": [
> {
> "name": "Version1",
> "type": "string"
> },
> {
> "name": "Version2",
> "type": "string"
> }
> ]
> }
> },
> {
> "name": "OtherData",
> "type": "string"
> }
> ]
> }
> {code}
> Say, node A knows only the first schema and node B knows the second schema,
> and the second schema has more fields.
> Any data generated by node B can be resolved by first schema 'cause the
> additional field is marked as skipped.
> But data generated by node A can not be resolved by second schema and throws
> an exception *"Don't know how to handle excess fields for reader."*
> This is because data is resolved exactly according to the auto-generated
> codec_traits which trying to read the excess field.
> The problem is we just can not only ignore the excess field in record, since
> the data after the troublesome record also needs to be resolved.
> Actually this problem stucked us for a very long time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1635:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pull request merged: [https://github.com/apache/avro/pull/301.] Thank you
[~wmatthews].
> C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive
> schema
>
>
> Key: AVRO-1635
> URL: https://issues.apache.org/jira/browse/AVRO-1635
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7
> Environment: Ubuntu 12.04, gcc 4.6.4
>Reporter: Heye Vöcking
>Assignee: William Matthews
>Priority: Major
> Labels: avro, c++, encode, recursive, schema
>
> Encoding an object with a recursive schema fails when using a jsonEncoder or
> a validatingEncoder. Here is an example:
> Output:
> {noformat}
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'
> what(): tr1::bad_weak_ptr
> {noformat}
> {code:title=container.json|borderStyle=solid}
> {
> "name": "Container",
> "doc": "Container to demonstrate the weak_ptr exception.",
> "type": "record",
> "fields": [{
> "name": "field",
> "type": {
> "name": "Object",
> "type": "record",
> "fields": [{
> "name": "value",
> "type": [
> "string",
> {"type": "map", "values": "Object"}
> ]
> }]
> }
> }]
> }
> {code}
> {code:title=example.cc|borderStyle=solid}
> #include
> #include
> #include
> #include
> #include
> #include "container.hh"
> int
> main()
> {
> std::ifstream ifs("container.json");
> avro::ValidSchema schema;
> avro::compileJsonSchema(ifs, schema);
> std::auto_ptr out = avro::memoryOutputStream();
> // Either one fails, here we use the jsonEncoder
> // avro::EncoderPtr encoder = avro::jsonEncoder(schema);
> avro::EncoderPtr encoder = avro::validatingEncoder(schema,
> avro::binaryEncoder());
>
> // An encoder that doesn't know the schema works fine
> // avro::EncoderPtr encoder = avro::binaryEncoder();
> encoder->init(*out);
> Container container;
> std::map object;
> // Note that it doesn't fail if we don't insert a value into the map
> object["a"].value.set_string("x");
> container.field.value.set_map(object);
> avro::encode(*encoder, container);
> return 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635889#comment-16635889
]
ASF subversion and git services commented on AVRO-1635:
---
Commit 389231ef31b02d444a73105ea4d6b5e0cbaac1d4 in avro's branch
refs/heads/master from Thiruvalluvan M G
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=389231e ]
Merge pull request #301 from wmatthews-google/fix-issue-1635
AVRO-1635 C++ schema aware encoders throw tr1::bad_weak_ptr exception for
recursive schema
> C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive
> schema
>
>
> Key: AVRO-1635
> URL: https://issues.apache.org/jira/browse/AVRO-1635
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7
> Environment: Ubuntu 12.04, gcc 4.6.4
>Reporter: Heye Vöcking
>Assignee: William Matthews
>Priority: Major
> Labels: avro, c++, encode, recursive, schema
>
> Encoding an object with a recursive schema fails when using a jsonEncoder or
> a validatingEncoder. Here is an example:
> Output:
> {noformat}
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'
> what(): tr1::bad_weak_ptr
> {noformat}
> {code:title=container.json|borderStyle=solid}
> {
> "name": "Container",
> "doc": "Container to demonstrate the weak_ptr exception.",
> "type": "record",
> "fields": [{
> "name": "field",
> "type": {
> "name": "Object",
> "type": "record",
> "fields": [{
> "name": "value",
> "type": [
> "string",
> {"type": "map", "values": "Object"}
> ]
> }]
> }
> }]
> }
> {code}
> {code:title=example.cc|borderStyle=solid}
> #include
> #include
> #include
> #include
> #include
> #include "container.hh"
> int
> main()
> {
> std::ifstream ifs("container.json");
> avro::ValidSchema schema;
> avro::compileJsonSchema(ifs, schema);
> std::auto_ptr out = avro::memoryOutputStream();
> // Either one fails, here we use the jsonEncoder
> // avro::EncoderPtr encoder = avro::jsonEncoder(schema);
> avro::EncoderPtr encoder = avro::validatingEncoder(schema,
> avro::binaryEncoder());
>
> // An encoder that doesn't know the schema works fine
> // avro::EncoderPtr encoder = avro::binaryEncoder();
> encoder->init(*out);
> Container container;
> std::map object;
> // Note that it doesn't fail if we don't insert a value into the map
> object["a"].value.set_string("x");
> container.field.value.set_map(object);
> avro::encode(*encoder, container);
> return 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635888#comment-16635888
]
ASF subversion and git services commented on AVRO-1635:
---
Commit 390a85409006f829d4d3564c2308c06ade211920 in avro's branch
refs/heads/master from [~wdm81]
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=390a854 ]
Fix https://issues.apache.org/jira/browse/AVRO-1635
- Use indirect symbols to "own" the production needed for records (so that
weak_ptrs have something to point to)
- Use a stack instead of a counter for the number of items in a repeated symbol
- A handful of tests.
> C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive
> schema
>
>
> Key: AVRO-1635
> URL: https://issues.apache.org/jira/browse/AVRO-1635
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7
> Environment: Ubuntu 12.04, gcc 4.6.4
>Reporter: Heye Vöcking
>Assignee: William Matthews
>Priority: Major
> Labels: avro, c++, encode, recursive, schema
>
> Encoding an object with a recursive schema fails when using a jsonEncoder or
> a validatingEncoder. Here is an example:
> Output:
> {noformat}
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'
> what(): tr1::bad_weak_ptr
> {noformat}
> {code:title=container.json|borderStyle=solid}
> {
> "name": "Container",
> "doc": "Container to demonstrate the weak_ptr exception.",
> "type": "record",
> "fields": [{
> "name": "field",
> "type": {
> "name": "Object",
> "type": "record",
> "fields": [{
> "name": "value",
> "type": [
> "string",
> {"type": "map", "values": "Object"}
> ]
> }]
> }
> }]
> }
> {code}
> {code:title=example.cc|borderStyle=solid}
> #include
> #include
> #include
> #include
> #include
> #include "container.hh"
> int
> main()
> {
> std::ifstream ifs("container.json");
> avro::ValidSchema schema;
> avro::compileJsonSchema(ifs, schema);
> std::auto_ptr out = avro::memoryOutputStream();
> // Either one fails, here we use the jsonEncoder
> // avro::EncoderPtr encoder = avro::jsonEncoder(schema);
> avro::EncoderPtr encoder = avro::validatingEncoder(schema,
> avro::binaryEncoder());
>
> // An encoder that doesn't know the schema works fine
> // avro::EncoderPtr encoder = avro::binaryEncoder();
> encoder->init(*out);
> Container container;
> std::map object;
> // Note that it doesn't fail if we don't insert a value into the map
> object["a"].value.set_string("x");
> container.field.value.set_map(object);
> avro::encode(*encoder, container);
> return 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635887#comment-16635887
]
ASF GitHub Bot commented on AVRO-1635:
--
thiru-apache closed pull request #301: AVRO-1635 C++ schema aware encoders
throw tr1::bad_weak_ptr exception for recursive schema
URL: https://github.com/apache/avro/pull/301
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c++/impl/parsing/JsonCodec.cc
b/lang/c++/impl/parsing/JsonCodec.cc
index ead44cab5..a7663a9e0 100644
--- a/lang/c++/impl/parsing/JsonCodec.cc
+++ b/lang/c++/impl/parsing/JsonCodec.cc
@@ -106,7 +106,7 @@ ProductionPtr JsonGrammarGenerator::doGenerate(const
NodePtr& n,
reverse(result->begin(), result->end());
m[n] = result;
-return result;
+return make_shared(1, Symbol::indirect(result));
}
case AVRO_ENUM:
{
@@ -363,6 +363,7 @@ template
size_t JsonDecoder::arrayStart()
{
parser_.advance(Symbol::sArrayStart);
+parser_.pushRepeatCount(0);
expect(JsonParser::tkArrayStart);
return arrayNext();
}
@@ -377,7 +378,7 @@ size_t JsonDecoder::arrayNext()
parser_.advance(Symbol::sArrayEnd);
return 0;
}
-parser_.setRepeatCount(1);
+parser_.nextRepeatCount(1);
return 1;
}
@@ -419,6 +420,7 @@ template
size_t JsonDecoder::mapStart()
{
parser_.advance(Symbol::sMapStart);
+parser_.pushRepeatCount(0);
expect(JsonParser::tkObjectStart);
return mapNext();
}
@@ -433,7 +435,7 @@ size_t JsonDecoder::mapNext()
parser_.advance(Symbol::sMapEnd);
return 0;
}
-parser_.setRepeatCount(1);
+parser_.nextRepeatCount(1);
return 1;
}
@@ -624,6 +626,7 @@ template
void JsonEncoder::arrayStart()
{
parser_.advance(Symbol::sArrayStart);
+parser_.pushRepeatCount(0);
out_.arrayStart();
}
@@ -639,6 +642,7 @@ template
void JsonEncoder::mapStart()
{
parser_.advance(Symbol::sMapStart);
+parser_.pushRepeatCount(0);
out_.objectStart();
}
@@ -653,7 +657,7 @@ void JsonEncoder::mapEnd()
template
void JsonEncoder::setItemCount(size_t count)
{
-parser_.setRepeatCount(count);
+parser_.nextRepeatCount(count);
}
template
diff --git a/lang/c++/impl/parsing/ResolvingDecoder.cc
b/lang/c++/impl/parsing/ResolvingDecoder.cc
index e0d25edfd..08518d93e 100644
--- a/lang/c++/impl/parsing/ResolvingDecoder.cc
+++ b/lang/c++/impl/parsing/ResolvingDecoder.cc
@@ -327,7 +327,7 @@ ProductionPtr ResolvingGrammarGenerator::doGenerate2(
m[key] = ProductionPtr();
ProductionPtr result = resolveRecords(writer, reader, m, m2);
m[key] = result;
-return result;
+ return make_shared(1, Symbol::indirect(result));
}
break;
@@ -636,11 +636,10 @@ size_t ResolvingDecoderImpl::arrayStart()
{
parser_.advance(Symbol::sArrayStart);
size_t result = base_->arrayStart();
+parser_.pushRepeatCount(result);
if (result == 0) {
parser_.popRepeater();
parser_.advance(Symbol::sArrayEnd);
-} else {
-parser_.setRepeatCount(result);
}
return result;
}
@@ -650,11 +649,10 @@ size_t ResolvingDecoderImpl::arrayNext()
{
parser_.processImplicitActions();
size_t result = base_->arrayNext();
+parser_.nextRepeatCount(result);
if (result == 0) {
parser_.popRepeater();
parser_.advance(Symbol::sArrayEnd);
-} else {
-parser_.setRepeatCount(result);
}
return result;
}
@@ -667,7 +665,7 @@ size_t ResolvingDecoderImpl::skipArray()
if (n == 0) {
parser_.pop();
} else {
-parser_.setRepeatCount(n);
+parser_.pushRepeatCount(n);
parser_.skip(*base_);
}
parser_.advance(Symbol::sArrayEnd);
@@ -679,11 +677,10 @@ size_t ResolvingDecoderImpl::mapStart()
{
parser_.advance(Symbol::sMapStart);
size_t result = base_->mapStart();
+parser_.pushRepeatCount(result);
if (result == 0) {
parser_.popRepeater();
parser_.advance(Symbol::sMapEnd);
-} else {
-parser_.setRepeatCount(result);
}
return result;
}
@@ -693,11 +690,10 @@ size_t ResolvingDecoderImpl::mapNext()
{
parser_.processImplicitActions();
size_t result = base_->mapNext();
+parser_.nextRepeatCount(result);
if (result == 0) {
parser_.popRepeater();
parser_.advance(Symbol::sMapEnd);
-} else {
-parser_.setRepeatCount(result);
}
return result;
}
@@ -710,7 +706,7 @@ size_t ResolvingDecoderImpl::skipMap()
if (n == 0) {
parser_.pop();
[jira] [Updated] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1635:
--
Status: Patch Available (was: Open)
> C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive
> schema
>
>
> Key: AVRO-1635
> URL: https://issues.apache.org/jira/browse/AVRO-1635
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7
> Environment: Ubuntu 12.04, gcc 4.6.4
>Reporter: Heye Vöcking
>Assignee: William Matthews
>Priority: Major
> Labels: avro, c++, encode, recursive, schema
>
> Encoding an object with a recursive schema fails when using a jsonEncoder or
> a validatingEncoder. Here is an example:
> Output:
> {noformat}
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'
> what(): tr1::bad_weak_ptr
> {noformat}
> {code:title=container.json|borderStyle=solid}
> {
> "name": "Container",
> "doc": "Container to demonstrate the weak_ptr exception.",
> "type": "record",
> "fields": [{
> "name": "field",
> "type": {
> "name": "Object",
> "type": "record",
> "fields": [{
> "name": "value",
> "type": [
> "string",
> {"type": "map", "values": "Object"}
> ]
> }]
> }
> }]
> }
> {code}
> {code:title=example.cc|borderStyle=solid}
> #include
> #include
> #include
> #include
> #include
> #include "container.hh"
> int
> main()
> {
> std::ifstream ifs("container.json");
> avro::ValidSchema schema;
> avro::compileJsonSchema(ifs, schema);
> std::auto_ptr out = avro::memoryOutputStream();
> // Either one fails, here we use the jsonEncoder
> // avro::EncoderPtr encoder = avro::jsonEncoder(schema);
> avro::EncoderPtr encoder = avro::validatingEncoder(schema,
> avro::binaryEncoder());
>
> // An encoder that doesn't know the schema works fine
> // avro::EncoderPtr encoder = avro::binaryEncoder();
> encoder->init(*out);
> Container container;
> std::map object;
> // Note that it doesn't fail if we don't insert a value into the map
> object["a"].value.set_string("x");
> container.field.value.set_map(object);
> avro::encode(*encoder, container);
> return 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-1635) C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive schema
[
https://issues.apache.org/jira/browse/AVRO-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. reassigned AVRO-1635:
-
Assignee: William Matthews
> C++ schema aware encoders throw tr1::bad_weak_ptr exception for recursive
> schema
>
>
> Key: AVRO-1635
> URL: https://issues.apache.org/jira/browse/AVRO-1635
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7
> Environment: Ubuntu 12.04, gcc 4.6.4
>Reporter: Heye Vöcking
>Assignee: William Matthews
>Priority: Major
> Labels: avro, c++, encode, recursive, schema
>
> Encoding an object with a recursive schema fails when using a jsonEncoder or
> a validatingEncoder. Here is an example:
> Output:
> {noformat}
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'
> what(): tr1::bad_weak_ptr
> {noformat}
> {code:title=container.json|borderStyle=solid}
> {
> "name": "Container",
> "doc": "Container to demonstrate the weak_ptr exception.",
> "type": "record",
> "fields": [{
> "name": "field",
> "type": {
> "name": "Object",
> "type": "record",
> "fields": [{
> "name": "value",
> "type": [
> "string",
> {"type": "map", "values": "Object"}
> ]
> }]
> }
> }]
> }
> {code}
> {code:title=example.cc|borderStyle=solid}
> #include
> #include
> #include
> #include
> #include
> #include "container.hh"
> int
> main()
> {
> std::ifstream ifs("container.json");
> avro::ValidSchema schema;
> avro::compileJsonSchema(ifs, schema);
> std::auto_ptr out = avro::memoryOutputStream();
> // Either one fails, here we use the jsonEncoder
> // avro::EncoderPtr encoder = avro::jsonEncoder(schema);
> avro::EncoderPtr encoder = avro::validatingEncoder(schema,
> avro::binaryEncoder());
>
> // An encoder that doesn't know the schema works fine
> // avro::EncoderPtr encoder = avro::binaryEncoder();
> encoder->init(*out);
> Container container;
> std::map object;
> // Note that it doesn't fail if we don't insert a value into the map
> object["a"].value.set_string("x");
> container.field.value.set_map(object);
> avro::encode(*encoder, container);
> return 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2186) avro::Exception string gives the incorrect message
[
https://issues.apache.org/jira/browse/AVRO-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-2186:
--
Status: Patch Available (was: Open)
Here is a pull request. [~Anubhav_CSG] please review.
> avro::Exception string gives the incorrect message
> --
>
> Key: AVRO-2186
> URL: https://issues.apache.org/jira/browse/AVRO-2186
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.2
> Environment: {code:java}
> Linux bream 2.6.39-200.24.1.el6uek.x86_64 #1 SMP Sat Jun 23 02:39:07 EDT 2012
> x86_64 x86_64 x86_64 GNU/Linux
> {code}
>
>Reporter: Anubhav Siddharth
>Assignee: Thiruvalluvan M. G.
>Priority: Critical
>
> When a field specified in schema is missing and we call avro::encode on the
> data. It throws an exception, which is as expected. But the issue is with the
> error message. It comes the other way round.
> Schema:
> {code:java}
> {
> "type" : "record",
> "name" : "userInfo",
> "fields" : [
> { "name" : "id", "type" : "int"},
> { "name" : "fullName", "type" : "string" }
> ]
> }
> {code}
> The generic datum to be encoded has no "id" provided. There is an empty
> GenericDatum for that field. Here is the excetion that is thrown:
> {code:java}
> avro::Exception caught: Invalid operation. Expected: Null got Int{code}
> The exception message should have been
> avro::Exception caught: Invalid operation. Expected: {color:#FF}*Int got
> Null*{color}
>
> {color:#33}Please let us know if you need more information on this.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-2186) avro::Exception string gives the incorrect message
[
https://issues.apache.org/jira/browse/AVRO-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. reassigned AVRO-2186:
-
Assignee: Thiruvalluvan M. G. (was: Goutham)
> avro::Exception string gives the incorrect message
> --
>
> Key: AVRO-2186
> URL: https://issues.apache.org/jira/browse/AVRO-2186
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.2
> Environment: {code:java}
> Linux bream 2.6.39-200.24.1.el6uek.x86_64 #1 SMP Sat Jun 23 02:39:07 EDT 2012
> x86_64 x86_64 x86_64 GNU/Linux
> {code}
>
>Reporter: Anubhav Siddharth
>Assignee: Thiruvalluvan M. G.
>Priority: Critical
>
> When a field specified in schema is missing and we call avro::encode on the
> data. It throws an exception, which is as expected. But the issue is with the
> error message. It comes the other way round.
> Schema:
> {code:java}
> {
> "type" : "record",
> "name" : "userInfo",
> "fields" : [
> { "name" : "id", "type" : "int"},
> { "name" : "fullName", "type" : "string" }
> ]
> }
> {code}
> The generic datum to be encoded has no "id" provided. There is an empty
> GenericDatum for that field. Here is the excetion that is thrown:
> {code:java}
> avro::Exception caught: Invalid operation. Expected: Null got Int{code}
> The exception message should have been
> avro::Exception caught: Invalid operation. Expected: {color:#FF}*Int got
> Null*{color}
>
> {color:#33}Please let us know if you need more information on this.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2220) std::bad_alloc when String or Bytes field has a negative length
[
https://issues.apache.org/jira/browse/AVRO-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-2220:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pull request merged after [~vimota] approved it:
https://github.com/apache/avro/pull/339
> std::bad_alloc when String or Bytes field has a negative length
> ---
>
> Key: AVRO-2220
> URL: https://issues.apache.org/jira/browse/AVRO-2220
> Project: Avro
> Issue Type: Bug
> Components: c++
>Reporter: Victor Mota
>Assignee: Victor Mota
>Priority: Major
> Attachments:
> poc-18e554fc65b937059584f21805da4b598f2266290f19d764da2c30ca1c829d0a (3)
>
>
> Attached is a sample file created by our Fuzzer running on the C++ library
> that causes an std::bad_alloc due to the string or byte field having an
> invalid negative integer length. The fix is trivial I'll send out a PR soon
> but it's something like:
>
> {code:java}
> void BinaryDecoder::decodeString(std::string& value)
> {
> // Preserve the sign to avoid allocating memory if len is negative.
> ssize_t len = decodeInt();
> if (len < 0) {
> throw Exception(
> boost::format("Cannot have a string of negative length: %1%") % len);
> }
> value.resize(len);
> if (len > 0) {
> in_.readBytes(reinterpret_cast([0]), len);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2220) std::bad_alloc when String or Bytes field has a negative length
[
https://issues.apache.org/jira/browse/AVRO-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635850#comment-16635850
]
ASF subversion and git services commented on AVRO-2220:
---
Commit 4505b498da9800f2d3143d7487ee07e99177808b in avro's branch
refs/heads/master from Thiruvalluvan M G
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=4505b49 ]
Merge pull request #339 from apache/AVRO-2220-handle-negative-lengths
AVRO-2220 Handled possible negative lengths in data in decoder
> std::bad_alloc when String or Bytes field has a negative length
> ---
>
> Key: AVRO-2220
> URL: https://issues.apache.org/jira/browse/AVRO-2220
> Project: Avro
> Issue Type: Bug
> Components: c++
>Reporter: Victor Mota
>Assignee: Victor Mota
>Priority: Major
> Attachments:
> poc-18e554fc65b937059584f21805da4b598f2266290f19d764da2c30ca1c829d0a (3)
>
>
> Attached is a sample file created by our Fuzzer running on the C++ library
> that causes an std::bad_alloc due to the string or byte field having an
> invalid negative integer length. The fix is trivial I'll send out a PR soon
> but it's something like:
>
> {code:java}
> void BinaryDecoder::decodeString(std::string& value)
> {
> // Preserve the sign to avoid allocating memory if len is negative.
> ssize_t len = decodeInt();
> if (len < 0) {
> throw Exception(
> boost::format("Cannot have a string of negative length: %1%") % len);
> }
> value.resize(len);
> if (len > 0) {
> in_.readBytes(reinterpret_cast([0]), len);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2220) std::bad_alloc when String or Bytes field has a negative length
[
https://issues.apache.org/jira/browse/AVRO-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635848#comment-16635848
]
ASF GitHub Bot commented on AVRO-2220:
--
thiru-apache closed pull request #339: AVRO-2220 Handled possible negative
lengths in data in decoder
URL: https://github.com/apache/avro/pull/339
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> std::bad_alloc when String or Bytes field has a negative length
> ---
>
> Key: AVRO-2220
> URL: https://issues.apache.org/jira/browse/AVRO-2220
> Project: Avro
> Issue Type: Bug
> Components: c++
>Reporter: Victor Mota
>Assignee: Victor Mota
>Priority: Major
> Attachments:
> poc-18e554fc65b937059584f21805da4b598f2266290f19d764da2c30ca1c829d0a (3)
>
>
> Attached is a sample file created by our Fuzzer running on the C++ library
> that causes an std::bad_alloc due to the string or byte field having an
> invalid negative integer length. The fix is trivial I'll send out a PR soon
> but it's something like:
>
> {code:java}
> void BinaryDecoder::decodeString(std::string& value)
> {
> // Preserve the sign to avoid allocating memory if len is negative.
> ssize_t len = decodeInt();
> if (len < 0) {
> throw Exception(
> boost::format("Cannot have a string of negative length: %1%") % len);
> }
> value.resize(len);
> if (len > 0) {
> in_.readBytes(reinterpret_cast([0]), len);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2224) Encode uniont types through std::any
[ https://issues.apache.org/jira/browse/AVRO-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-2224. --- Resolution: Duplicate This is the duplicate of AVRO-2037. > Encode uniont types through std::any > > > Key: AVRO-2224 > URL: https://issues.apache.org/jira/browse/AVRO-2224 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: Rui Maciel >Priority: Major > > With the release of C++17, standard C\+\+ started supporting > [std::any|https://en.cppreference.com/w/cpp/utility/any]. It would be > awesome if Apache Avro could generate C++ code that expressed union types > through std::any instead of providing an ad-hoc implementation, which would > enable using Avro-generated code in a more idiomatic way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2153) Generating C++ code with avrogencpp produces unclear error message
[ https://issues.apache.org/jira/browse/AVRO-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-2153: -- Resolution: Cannot Reproduce Status: Resolved (was: Patch Available) The problem is specific to the reporter's environment and the reporter has closed the pull request. > Generating C++ code with avrogencpp produces unclear error message > -- > > Key: AVRO-2153 > URL: https://issues.apache.org/jira/browse/AVRO-2153 > Project: Avro > Issue Type: Bug > Components: c++ >Reporter: Nick Byman >Priority: Minor > > If you put a bad name for a json node in an avro file, you can sometimes get: > "Exception: boost::too_many_args: format-string referred", when avro is > trying to tell you the name of the bad json node, because the node's name > isn't passed to boost::format correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2202) Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when combined with musl libc static builds
[ https://issues.apache.org/jira/browse/AVRO-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-2202: -- Status: Patch Available (was: Open) > Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when > combined with musl libc static builds > > > Key: AVRO-2202 > URL: https://issues.apache.org/jira/browse/AVRO-2202 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 > Environment: OS: Gentoo Linux > Compiler: Gcc 6.4.0 targeting musl libc 1.19 with stdlibc++ in a static build >Reporter: Josh Scoggins >Priority: Major > > Observed with master and version 1.8.2. Inside of the C++ implementation's > CMakeLists.txt the lines 56-60 were added sometime in 2016 which activate > features inside of libstdc++ to help find errors. This change has two side > effects: > # -g is eliminated from the build so debugging gdb becomes harder as source > position data has been removed > # The safety containers that implicitly wrap all standard containers somehow > (I can't figure out exactly how) cause memory corruption when setMetadata is > called (in version 1.8.2) or when a Writer object is destroyed (on master). > Since our software uses musl libc with libstdc++ I believe that it is an > interaction with these debugging features which cause memory corruption (as I > said, I have no clue _why_ it is happening, only that these features _cause_ > it :() > Previously, we were using 1.7.7 which did not have these lines in its > CMakeLists.txt and thus did not exhibit any sort of memory corruption. I was > able to work around this by removing lines 56-60 from CMakeLists.txt so our > static libs could be built and our unit tests not crash by writing to memory > it did not own. There should be a toggle or something similar to turn off > these features for debug builds as they are for developers of avro only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2220) std::bad_alloc when String or Bytes field has a negative length
[
https://issues.apache.org/jira/browse/AVRO-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635812#comment-16635812
]
Thiruvalluvan M. G. commented on AVRO-2220:
---
Thank you [~vimota] for pointing out the issue. I created a pull request
[https://github.com/apache/avro/pull/339.] Please review.
> std::bad_alloc when String or Bytes field has a negative length
> ---
>
> Key: AVRO-2220
> URL: https://issues.apache.org/jira/browse/AVRO-2220
> Project: Avro
> Issue Type: Bug
> Components: c++
>Reporter: Victor Mota
>Assignee: Victor Mota
>Priority: Major
> Attachments:
> poc-18e554fc65b937059584f21805da4b598f2266290f19d764da2c30ca1c829d0a (3)
>
>
> Attached is a sample file created by our Fuzzer running on the C++ library
> that causes an std::bad_alloc due to the string or byte field having an
> invalid negative integer length. The fix is trivial I'll send out a PR soon
> but it's something like:
>
> {code:java}
> void BinaryDecoder::decodeString(std::string& value)
> {
> // Preserve the sign to avoid allocating memory if len is negative.
> ssize_t len = decodeInt();
> if (len < 0) {
> throw Exception(
> boost::format("Cannot have a string of negative length: %1%") % len);
> }
> value.resize(len);
> if (len > 0) {
> in_.readBytes(reinterpret_cast([0]), len);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2209) Default value type validation over-zealous
[
https://issues.apache.org/jira/browse/AVRO-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-2209:
--
Status: Patch Available (was: Open)
> Default value type validation over-zealous
> --
>
> Key: AVRO-2209
> URL: https://issues.apache.org/jira/browse/AVRO-2209
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.2
>Reporter: Darryl Green
>Priority: Major
>
> From the Avro specification re default values (and hence JSON encoding in
> general):
>
> |field default values|
> ||avro type||json type||example||
> |null|null|null|
> |boolean|boolean|true|
> |int,long|integer|1|
> |float,double|number|1.1|
> |bytes|string|"\u00FF"|
> |string|string|"foo"|
> |record|object|{"a": 1}|
> |enum|string|"FOO"|
> |array|array|[1]|
> |map|object|{"a": 1}|
> |fixed|string|"\u00ff"|
>
> Note that float and double have a "json type" of number (while int, long have
> a "json type" of integer. In JSON an integer is a number that is constrained
> to be an integer. There is no way to deduce from a JSON value that has no
> fractional part whether that value is a number or an integer - it is
> either/both.
> I believe that the following schema is, on that basis, valid:
> "{ \"name\":\"test\", \"type\": \"record\", \"fields\": [
> {\"name\": \"double\",\"type\": \"double\",\"default\" : 2 }
> ]}",
> We have a substantial body of similar schema in use but have not attempted
> to use C++ to resolve them before - and now this is failing.
> Fix is reasonably straight forward - PR with tests:
> [https://github.com/apache/avro/pull/326]
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2229) Ability to test using a Docker image
[ https://issues.apache.org/jira/browse/AVRO-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635788#comment-16635788 ] ASF GitHub Bot commented on AVRO-2229: -- Fokko commented on issue #336: [AVRO-2229] Test using a Docker image URL: https://github.com/apache/avro/pull/336#issuecomment-426340029 @kojiromike This is actually a valid point of the rat checker. The licenses need to be set correctly I think I'll integrate the `run-tests.sh` in the root `build.sh`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ability to test using a Docker image > > > Key: AVRO-2229 > URL: https://issues.apache.org/jira/browse/AVRO-2229 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Fokko Driesprong >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2222) Avro C++ documentation is missing code snippets
[ https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thank you [~da77a] for the patch. I created a Pull Request and merged: https://github.com/apache/avro/pull/338 > Avro C++ documentation is missing code snippets > --- > > Key: AVRO- > URL: https://issues.apache.org/jira/browse/AVRO- > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4 >Reporter: Rui Maciel >Assignee: Darryl Green >Priority: Major > Attachments: > 0001-AVRO--Avro-C-documentation-is-missing-code-snipp.patch > > > The Apache Avro C++ Documentation page[¹] includes an example on how to use > Avro with C++ but unfortunately most code snippets are missing, which renders > the document unusable. > > [¹]http://avro.apache.org/docs/1.8.2/api/cpp/html/index.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2222) Avro C++ documentation is missing code snippets
[ https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635777#comment-16635777 ] ASF subversion and git services commented on AVRO-: --- Commit a943df760f6df520702a54e9b061e98250172c8e in avro's branch refs/heads/master from Thiruvalluvan M G [ https://gitbox.apache.org/repos/asf?p=avro.git;h=a943df7 ] Merge pull request #338 from apache/AVRO--doxygen-fix Fixed doxygen > Avro C++ documentation is missing code snippets > --- > > Key: AVRO- > URL: https://issues.apache.org/jira/browse/AVRO- > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4 >Reporter: Rui Maciel >Assignee: Darryl Green >Priority: Major > Attachments: > 0001-AVRO--Avro-C-documentation-is-missing-code-snipp.patch > > > The Apache Avro C++ Documentation page[¹] includes an example on how to use > Avro with C++ but unfortunately most code snippets are missing, which renders > the document unusable. > > [¹]http://avro.apache.org/docs/1.8.2/api/cpp/html/index.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2219) std::bad_alloc when String or Bytes field has a negative length
[
https://issues.apache.org/jira/browse/AVRO-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. resolved AVRO-2219.
---
Resolution: Duplicate
Duplicate of AVRO-2220
> std::bad_alloc when String or Bytes field has a negative length
> ---
>
> Key: AVRO-2219
> URL: https://issues.apache.org/jira/browse/AVRO-2219
> Project: Avro
> Issue Type: Bug
> Components: c++
>Reporter: Victor Mota
>Assignee: Victor Mota
>Priority: Major
> Attachments:
> poc-18e554fc65b937059584f21805da4b598f2266290f19d764da2c30ca1c829d0a (3)
>
>
> Attached is a sample file created by our Fuzzer running on the C++ library
> that causes an std::bad_alloc due to the string or byte field having an
> invalid negative integer length. The fix is trivial I'll send out a PR soon
> but it's something like:
>
> {code:java}
> void BinaryDecoder::decodeString(std::string& value)
> {
> // Preserve the sign to avoid allocating memory if len is negative.
> ssize_t len = decodeInt();
> if (len < 0) {
> throw Exception(
> boost::format("Cannot have a string of negative length: %1%") % len);
> }
> value.resize(len);
> if (len > 0) {
> in_.readBytes(reinterpret_cast([0]), len);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2213) C++ tests fail with boost 1.67+
[ https://issues.apache.org/jira/browse/AVRO-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-2213. --- Resolution: Fixed Pull request merged: https://github.com/apache/avro/pull/328 > C++ tests fail with boost 1.67+ > --- > > Key: AVRO-2213 > URL: https://issues.apache.org/jira/browse/AVRO-2213 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Assignee: William Matthews >Priority: Minor > > Boost 1.67 and later now returns an error when multiple tests are added with > the same name > ([https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/change_log.html).] > This fails for 3 tests: > CodecTests.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'testCodecResolving2_0' registered > multiple times > DataFileTests.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'DataFileTest__testReadFull' registered multiple times > unittest.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'T__test' registered multiple times > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2213) C++ tests fail with boost 1.67+
[ https://issues.apache.org/jira/browse/AVRO-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-2213: - Assignee: William Matthews > C++ tests fail with boost 1.67+ > --- > > Key: AVRO-2213 > URL: https://issues.apache.org/jira/browse/AVRO-2213 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Assignee: William Matthews >Priority: Minor > > Boost 1.67 and later now returns an error when multiple tests are added with > the same name > ([https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/change_log.html).] > This fails for 3 tests: > CodecTests.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'testCodecResolving2_0' registered > multiple times > DataFileTests.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'DataFileTest__testReadFull' registered multiple times > unittest.cc: > Test setup error: boost::unit_test::framework::setup_error: test unit with > name 'T__test' registered multiple times > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2214) Support sync and seek in C++ DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-2214. --- Resolution: Fixed Merged pull request https://github.com/apache/avro/pull/328 > Support sync and seek in C++ DataFileReader > --- > > Key: AVRO-2214 > URL: https://issues.apache.org/jira/browse/AVRO-2214 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Assignee: William Matthews >Priority: Minor > > Java DataFileReader supports sync, seek, pastSync, etc. which allow parallel > reads of files, and reasonably efficient "tailing" of files. It would be > great if these were supported in C++ too. > Also, I think this would serve as a bit of a workaround for > https://issues.apache.org/jira/browse/AVRO-2178 (stat a file & see if it has > grown, sync/seek, read, repeat). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2214) Support sync and seek in C++ DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-2214: - Assignee: William Matthews > Support sync and seek in C++ DataFileReader > --- > > Key: AVRO-2214 > URL: https://issues.apache.org/jira/browse/AVRO-2214 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Assignee: William Matthews >Priority: Minor > > Java DataFileReader supports sync, seek, pastSync, etc. which allow parallel > reads of files, and reasonably efficient "tailing" of files. It would be > great if these were supported in C++ too. > Also, I think this would serve as a bit of a workaround for > https://issues.apache.org/jira/browse/AVRO-2178 (stat a file & see if it has > grown, sync/seek, read, repeat). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2214) Support sync and seek in C++ DataFileReader
[
https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635734#comment-16635734
]
ASF GitHub Bot commented on AVRO-2214:
--
thiru-apache closed pull request #328: AVRO-2214 Support sync and seek in C++
DataFileReader
URL: https://github.com/apache/avro/pull/328
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c++/api/DataFile.hh b/lang/c++/api/DataFile.hh
index bff309770..4236d3537 100644
--- a/lang/c++/api/DataFile.hh
+++ b/lang/c++/api/DataFile.hh
@@ -172,12 +172,14 @@ public:
*/
class AVRO_DECL DataFileReaderBase : boost::noncopyable {
const std::string filename_;
-const std::auto_ptr stream_;
+const std::auto_ptr stream_;
const DecoderPtr decoder_;
int64_t objectCount_;
bool eof_;
Codec codec_;
-
+int64_t blockStart_;
+int64_t blockEnd_;
+
ValidSchema readerSchema_;
ValidSchema dataSchema_;
DecoderPtr dataDecoder_;
@@ -247,6 +249,29 @@ public:
* Closes the reader. No further operation is possible on this reader.
*/
void close();
+
+/**
+ * Move to a specific, known synchronization point, for example one
returned
+ * from tell() after sync().
+ */
+void seek(int64_t position);
+
+/**
+ * Move to the next synchronization point after a position. To process a
+ * range of file entires, call this with the starting position, then check
+ * pastSync() with the end point before each use of decoder().
+ */
+void sync(int64_t position);
+
+/**
+ * Return true if past the next synchronization point after a position.
+ */
+bool pastSync(int64_t position);
+
+/**
+ * Return the last synchronization point before our current position.
+ */
+int64_t previousSync();
};
/**
@@ -330,6 +355,29 @@ public:
* Closes the reader. No further operation is possible on this reader.
*/
void close() { return base_->close(); }
+
+/**
+ * Move to a specific, known synchronization point, for example one
returned
+ * from previousSync().
+ */
+void seek(int64_t position) { base_->seek(position); }
+
+/**
+ * Move to the next synchronization point after a position. To process a
+ * range of file entires, call this with the starting position, then check
+ * pastSync() with the end point before each call to read().
+ */
+void sync(int64_t position) { base_->sync(position); }
+
+/**
+ * Return true if past the next synchronization point after a position.
+ */
+bool pastSync(int64_t position) { return base_->pastSync(position); }
+
+/**
+ * Return the last synchronization point before our current position.
+ */
+int64_t previousSync() { return base_->previousSync(); }
};
} // namespace avro
diff --git a/lang/c++/api/Stream.hh b/lang/c++/api/Stream.hh
index 92b2334d2..42ccf0a00 100644
--- a/lang/c++/api/Stream.hh
+++ b/lang/c++/api/Stream.hh
@@ -75,6 +75,31 @@ public:
virtual size_t byteCount() const = 0;
};
+/**
+ * An InputStream which also supports seeking to a specific offset.
+ */
+class AVRO_DECL SeekableInputStream : public InputStream {
+protected:
+
+/**
+ * An empty constuctor.
+ */
+SeekableInputStream() { }
+
+public:
+/**
+ * Destructor.
+ */
+virtual ~SeekableInputStream() { }
+
+/**
+ * Seek to a specific position in the stream. This may invalidate pointers
+ * returned from next(). This will also reset byteCount() to the given
+ * position.
+ */
+virtual void seek(int64_t position) = 0;
+};
+
/**
* A no-copy output stream.
*/
@@ -161,8 +186,10 @@ AVRO_DECL std::auto_ptr
fileOutputStream(const char* filename,
* Returns a new InputStream whose contents come from the given file.
* Data is read in chunks of given buffer size.
*/
-AVRO_DECL std::auto_ptr fileInputStream(const char* filename,
-size_t bufferSize = 8 * 1024);
+AVRO_DECL std::auto_ptr fileInputStream(
+const char *filename, size_t bufferSize = 8 * 1024);
+AVRO_DECL std::auto_ptr fileSeekableInputStream(
+const char *filename, size_t bufferSize = 8 * 1024);
/**
* Returns a new OutputStream whose contents will be sent to the given
@@ -177,8 +204,8 @@ AVRO_DECL std::auto_ptr
ostreamOutputStream(std::ostream& os,
* std::istream. The std::istream object should outlive the returned
* InputStream.
*/
-AVRO_DECL std::auto_ptr istreamInputStream(std::istream& in,
-size_t bufferSize = 8 * 1024);
+AVRO_DECL std::auto_ptr istreamInputStream(
+std::istream , size_t bufferSize = 8 * 1024);
/** A convenience class for reading from an InputStream */
struct
[jira] [Commented] (AVRO-2213) C++ tests fail with boost 1.67+
[
https://issues.apache.org/jira/browse/AVRO-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635736#comment-16635736
]
ASF GitHub Bot commented on AVRO-2213:
--
thiru-apache closed pull request #327: AVRO-2213 C++ tests fail with boost 1.67+
URL: https://github.com/apache/avro/pull/327
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c++/test/CodecTests.cc b/lang/c++/test/CodecTests.cc
index f8bbe84d0..064351b3e 100644
--- a/lang/c++/test/CodecTests.cc
+++ b/lang/c++/test/CodecTests.cc
@@ -1363,9 +1363,17 @@ static const TestData4 data4BinaryOnly[] = {
#define COUNTOF(x) sizeof(x) / sizeof(x[0])
#define ENDOF(x)(x) + COUNTOF(x)
-#define ADD_TESTS(testSuite, Factory, testFunc, data) \
-testSuite.add(BOOST_PARAM_TEST_CASE(, \
-data, data + COUNTOF(data)))
+// Boost 1.67 and later expects test cases to have unique names. This dummy
+// helper functions leads to names which compose 'testFunc', 'Factory', and
+// 'data'.
+template
+Test testWithData(const Test , const Data ) {
+boost::ignore_unused(data);
+return test;
+}
+#define ADD_TESTS(testSuite, Factory, testFunc, data) \
+testSuite.add(BOOST_PARAM_TEST_CASE( \
+testWithData(, data), data, data + COUNTOF(data)))
struct BinaryEncoderFactory {
static EncoderPtr newEncoder(const ValidSchema& schema) {
diff --git a/lang/c++/test/DataFileTests.cc b/lang/c++/test/DataFileTests.cc
index 27a7ce9ca..1b1687af5 100644
--- a/lang/c++/test/DataFileTests.cc
+++ b/lang/c++/test/DataFileTests.cc
@@ -498,40 +498,67 @@ void addReaderTests(test_suite* ts, const
shared_ptr& t)
test_suite*
init_unit_test_suite( int argc, char* argv[] )
{
-test_suite* ts= BOOST_TEST_SUITE("DataFile tests");
-shared_ptr t1(new DataFileTest("test1.df", sch, isch));
-ts->add(BOOST_CLASS_TEST_CASE(::testWrite, t1));
-addReaderTests(ts, t1);
-
-shared_ptr t2(new DataFileTest("test2.df", sch, isch));
-ts->add(BOOST_CLASS_TEST_CASE(::testWriteGeneric, t2));
-addReaderTests(ts, t2);
-
-shared_ptr t3(new DataFileTest("test3.df", dsch, dblsch));
-ts->add(BOOST_CLASS_TEST_CASE(::testWriteDouble, t3));
-ts->add(BOOST_CLASS_TEST_CASE(::testReadDouble, t3));
-ts->add(BOOST_CLASS_TEST_CASE(::testReadDoubleTwoStep, t3));
-ts->add(BOOST_CLASS_TEST_CASE(::testReadDoubleTwoStepProject,
-t3));
-ts->add(BOOST_CLASS_TEST_CASE(::testCleanup, t3));
-
-shared_ptr t4(new DataFileTest("test4.df", dsch, dblsch));
-ts->add(BOOST_CLASS_TEST_CASE(::testTruncate, t4));
-ts->add(BOOST_CLASS_TEST_CASE(::testCleanup, t4));
-
-shared_ptr t5(new DataFileTest("test5.df", sch, isch));
-ts->add(BOOST_CLASS_TEST_CASE(::testWriteGenericByName, t5));
-addReaderTests(ts, t5);
-
-shared_ptr t6(new DataFileTest("test6.df", dsch, dblsch));
-ts->add(BOOST_CLASS_TEST_CASE(::testZip, t6));
-shared_ptr t8(new DataFileTest("test8.df", dsch, dblsch));
+{
+test_suite *ts = BOOST_TEST_SUITE("DataFile tests: test1.df");
+shared_ptr t1(new DataFileTest("test1.df", sch, isch));
+ts->add(BOOST_CLASS_TEST_CASE(::testWrite, t1));
+addReaderTests(ts, t1);
+boost::unit_test::framework::master_test_suite().add(ts);
+}
+{
+test_suite *ts = BOOST_TEST_SUITE("DataFile tests: test2.df");
+shared_ptr t2(new DataFileTest("test2.df", sch, isch));
+ts->add(BOOST_CLASS_TEST_CASE(::testWriteGeneric, t2));
+addReaderTests(ts, t2);
+boost::unit_test::framework::master_test_suite().add(ts);
+}
+{
+test_suite *ts = BOOST_TEST_SUITE("DataFile tests: test3.df");
+shared_ptr t3(new DataFileTest("test3.df", dsch,
dblsch));
+ts->add(BOOST_CLASS_TEST_CASE(::testWriteDouble, t3));
+ts->add(BOOST_CLASS_TEST_CASE(::testReadDouble, t3));
+ts->add(
+BOOST_CLASS_TEST_CASE(::testReadDoubleTwoStep, t3));
+ts->add(BOOST_CLASS_TEST_CASE(
+::testReadDoubleTwoStepProject, t3));
+ts->add(BOOST_CLASS_TEST_CASE(::testCleanup, t3));
+boost::unit_test::framework::master_test_suite().add(ts);
+}
+{
+test_suite *ts = BOOST_TEST_SUITE("DataFile tests: test4.df");
+shared_ptr t4(new DataFileTest("test4.df", dsch,
dblsch));
+ts->add(BOOST_CLASS_TEST_CASE(::testTruncate, t4));
+ts->add(BOOST_CLASS_TEST_CASE(::testCleanup, t4));
+boost::unit_test::framework::master_test_suite().add(ts);
+}
+{
+test_suite *ts = BOOST_TEST_SUITE("DataFile tests: test5.df");
+shared_ptr t5(new DataFileTest("test5.df", sch, isch));
+ts->add(
+
[jira] [Commented] (AVRO-2214) Support sync and seek in C++ DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635735#comment-16635735 ] ASF subversion and git services commented on AVRO-2214: --- Commit 3efbd4c1a5d81966ab5dd01b394c4b0a188bf871 in avro's branch refs/heads/master from Thiruvalluvan M G [ https://gitbox.apache.org/repos/asf?p=avro.git;h=3efbd4c ] Merge pull request #328 from wmatthews-google/avro-2214-support-seek AVRO-2214 Support sync and seek in C++ DataFileReader > Support sync and seek in C++ DataFileReader > --- > > Key: AVRO-2214 > URL: https://issues.apache.org/jira/browse/AVRO-2214 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Priority: Minor > > Java DataFileReader supports sync, seek, pastSync, etc. which allow parallel > reads of files, and reasonably efficient "tailing" of files. It would be > great if these were supported in C++ too. > Also, I think this would serve as a bit of a workaround for > https://issues.apache.org/jira/browse/AVRO-2178 (stat a file & see if it has > grown, sync/seek, read, repeat). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2214) Support sync and seek in C++ DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-2214: - Assignee: Thiruvalluvan M. G. > Support sync and seek in C++ DataFileReader > --- > > Key: AVRO-2214 > URL: https://issues.apache.org/jira/browse/AVRO-2214 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Assignee: Thiruvalluvan M. G. >Priority: Minor > > Java DataFileReader supports sync, seek, pastSync, etc. which allow parallel > reads of files, and reasonably efficient "tailing" of files. It would be > great if these were supported in C++ too. > Also, I think this would serve as a bit of a workaround for > https://issues.apache.org/jira/browse/AVRO-2178 (stat a file & see if it has > grown, sync/seek, read, repeat). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2214) Support sync and seek in C++ DataFileReader
[ https://issues.apache.org/jira/browse/AVRO-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. reassigned AVRO-2214: - Assignee: (was: Thiruvalluvan M. G.) > Support sync and seek in C++ DataFileReader > --- > > Key: AVRO-2214 > URL: https://issues.apache.org/jira/browse/AVRO-2214 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: William Matthews >Priority: Minor > > Java DataFileReader supports sync, seek, pastSync, etc. which allow parallel > reads of files, and reasonably efficient "tailing" of files. It would be > great if these were supported in C++ too. > Also, I think this would serve as a bit of a workaround for > https://issues.apache.org/jira/browse/AVRO-2178 (stat a file & see if it has > grown, sync/seek, read, repeat). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635558#comment-16635558
]
ASF GitHub Bot commented on AVRO-2079:
--
pvorb commented on issue #248: AVRO-2079: Add ability to generate Java8 native
date/time classes
URL: https://github.com/apache/avro/pull/248#issuecomment-426295060
I have an account, my username's also pvorb over there, but I have no
permissions to change the assignee of that ticket. You can assign me if you
like.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635515#comment-16635515
]
ASF GitHub Bot commented on AVRO-2079:
--
nandorKollar closed pull request #248: AVRO-2079: Add ability to generate Java8
native date/time classes
URL: https://github.com/apache/avro/pull/248
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/lang/java/avro/src/main/java/org/apache/avro/data/Java8TimeConversions.java
b/lang/java/avro/src/main/java/org/apache/avro/data/Java8TimeConversions.java
new file mode 100644
index 0..b172bd887
--- /dev/null
+++
b/lang/java/avro/src/main/java/org/apache/avro/data/Java8TimeConversions.java
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.avro.data;
+
+import org.apache.avro.Conversion;
+import org.apache.avro.LogicalType;
+import org.apache.avro.Schema;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalTime;
+import java.util.concurrent.TimeUnit;
+
+public class Java8TimeConversions {
+ public static class DateConversion extends Conversion {
+
+@Override
+public Class getConvertedType() {
+ return LocalDate.class;
+}
+
+@Override
+public String getLogicalTypeName() {
+ return "date";
+}
+
+@Override
+public LocalDate fromInt(Integer daysFromEpoch, Schema schema, LogicalType
type) {
+ return LocalDate.ofEpochDay(daysFromEpoch);
+}
+
+@Override
+public Integer toInt(LocalDate date, Schema schema, LogicalType type) {
+ long epochDays = date.toEpochDay();
+
+ return Math.toIntExact(epochDays);
+}
+ }
+
+ public static class TimeMillisConversion extends Conversion {
+@Override
+public Class getConvertedType() {
+ return LocalTime.class;
+}
+
+@Override
+public String getLogicalTypeName() {
+ return "time-millis";
+}
+
+@Override
+public LocalTime fromInt(Integer millisFromMidnight, Schema schema,
LogicalType type) {
+ return
LocalTime.ofNanoOfDay(TimeUnit.MILLISECONDS.toNanos(millisFromMidnight));
+}
+
+@Override
+public Integer toInt(LocalTime time, Schema schema, LogicalType type) {
+ return
Math.toIntExact(TimeUnit.NANOSECONDS.toMillis(time.toNanoOfDay()));
+}
+ }
+
+ public static class TimeMicrosConversion extends Conversion {
+@Override
+public Class getConvertedType() {
+ return LocalTime.class;
+}
+
+@Override
+public String getLogicalTypeName() {
+ return "time-micros";
+}
+
+@Override
+public LocalTime fromLong(Long microsFromMidnight, Schema schema,
LogicalType type) {
+ return
LocalTime.ofNanoOfDay(TimeUnit.MICROSECONDS.toNanos(microsFromMidnight));
+}
+
+@Override
+public Long toLong(LocalTime time, Schema schema, LogicalType type) {
+ return TimeUnit.NANOSECONDS.toMicros(time.toNanoOfDay());
+}
+ }
+
+ public static class TimestampMillisConversion extends Conversion {
+@Override
+public Class getConvertedType() {
+ return Instant.class;
+}
+
+@Override
+public String getLogicalTypeName() {
+ return "timestamp-millis";
+}
+
+@Override
+public Instant fromLong(Long millisFromEpoch, Schema schema, LogicalType
type) {
+ return Instant.ofEpochMilli(millisFromEpoch);
+}
+
+@Override
+public Long toLong(Instant timestamp, Schema schema, LogicalType type) {
+ return timestamp.toEpochMilli();
+}
+ }
+
+ public static class TimestampMicrosConversion extends Conversion {
+@Override
+public Class getConvertedType() {
+ return Instant.class;
+}
+
+@Override
+public String getLogicalTypeName() {
+ return "timestamp-micros";
+}
+
+@Override
+public Instant fromLong(Long microsFromEpoch, Schema schema, LogicalType
type) {
+ long epochSeconds = microsFromEpoch / (1_000_000);
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635514#comment-16635514
]
ASF GitHub Bot commented on AVRO-2079:
--
nandorKollar commented on issue #248: AVRO-2079: Add ability to generate Java8
native date/time classes
URL: https://github.com/apache/avro/pull/248#issuecomment-426280537
Sure I'll. @pvorb could you please assign AVRO-2079 to yourself, if you have
an Apache Jira account?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635442#comment-16635442
]
ASF GitHub Bot commented on AVRO-2079:
--
pvorb commented on issue #248: AVRO-2079: Add ability to generate Java8 native
date/time classes
URL: https://github.com/apache/avro/pull/248#issuecomment-426264250
@nandorKollar Since you merged #309, could you close this old pull request?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635439#comment-16635439
]
ASF GitHub Bot commented on AVRO-2079:
--
pvorb commented on issue #309: AVRO-2079: Add ability to generate Java8 native
date/time classes (new)
URL: https://github.com/apache/avro/pull/309#issuecomment-426263732
Thanks, @nandorKollar!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
