[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693973#comment-16693973
]
ASF subversion and git services commented on AVRO-1658:
---
Commit 619c6c0bf5b3440cfe80fff8b1c399b91d1d69db in avro's branch
refs/heads/master from rstata
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=619c6c0 ]
AVRO-1658 Added Javadoc to AvroDoc.java, which was missing on initial commit.
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693972#comment-16693972
]
ASF GitHub Bot commented on AVRO-1658:
--
dkulp closed pull request #388: AVRO-1658 Added license to AvroDoc.java
URL: https://github.com/apache/avro/pull/388
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
b/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
index 7b46a476b..7226a02cc 100644
--- a/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
+++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.avro.reflect;
import java.lang.annotation.ElementType;
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2250) Release 1.9.0
[ https://issues.apache.org/jira/browse/AVRO-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693971#comment-16693971 ] Daniel Kulp commented on AVRO-2250: --- Not sure yet... we're slowly going through almost 2 years of open pull requests. It's taking time, but at least we're moving forward. > Release 1.9.0 > - > > Key: AVRO-2250 > URL: https://issues.apache.org/jira/browse/AVRO-2250 > Project: Apache Avro > Issue Type: Task >Reporter: Nandor Kollar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymie Stata reopened AVRO-1658:
Assignee: Raymie Stata (was: Zhaonan Sun)
The file {{AvroDoc.java}} doesn't have a license, cause the build to break
(grumble). Will send a patch for this shortly.
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693948#comment-16693948
]
ASF GitHub Bot commented on AVRO-1658:
--
rstata opened a new pull request #388: AVRO-1658 Added license to AvroDoc.java
URL: https://github.com/apache/avro/pull/388
License file for org.apache.avro.reflect.AvroDoc was missing a license,
breaking the build. This patch adds the standard lic header.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1658:
--
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Zhaonan Sun
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693754#comment-16693754
]
ASF GitHub Bot commented on AVRO-1658:
--
dkulp closed pull request #189: AVRO-1658: Java: Add reflection annotation
@AvroDoc.
URL: https://github.com/apache/avro/pull/189
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
b/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
new file mode 100644
index 0..7b46a476b
--- /dev/null
+++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/AvroDoc.java
@@ -0,0 +1,17 @@
+package org.apache.avro.reflect;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Sets the avrodoc for this java field.
+ * When reading into this class, a reflectdatumreader
+ * looks for a schema field with the avrodoc.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.TYPE, ElementType.FIELD})
+public @interface AvroDoc {
+ String value();
+}
diff --git
a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
index eb9f5fb0f..29f2acf71 100644
--- a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
+++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
@@ -572,6 +572,8 @@ protected Schema createSchema(Type type, Map
names) {
String fullName = c.getName();
Schema schema = names.get(fullName);
if (schema == null) {
+AvroDoc annotatedDoc = c.getAnnotation(AvroDoc.class);// Docstring
+String doc = (annotatedDoc != null) ? annotatedDoc.value() : null;
String name = c.getSimpleName();
String space = c.getPackage() == null ? "" : c.getPackage().getName();
if (c.getEnclosingClass() != null) // nested class
@@ -588,18 +590,18 @@ protected Schema createSchema(Type type,
Map names) {
Enum[] constants = (Enum[])c.getEnumConstants();
for (int i = 0; i < constants.length; i++)
symbols.add(constants[i].name());
- schema = Schema.createEnum(name, null /* doc */, space, symbols);
+ schema = Schema.createEnum(name, doc, space, symbols);
consumeAvroAliasAnnotation(c, schema);
} else if (GenericFixed.class.isAssignableFrom(c)) { // fixed
int size = c.getAnnotation(FixedSize.class).value();
- schema = Schema.createFixed(name, null /* doc */, space, size);
+ schema = Schema.createFixed(name, doc, space, size);
consumeAvroAliasAnnotation(c, schema);
} else if (IndexedRecord.class.isAssignableFrom(c)) { // specific
return super.createSchema(type, names);
} else { // record
List fields = new ArrayList();
boolean error = Throwable.class.isAssignableFrom(c);
- schema = Schema.createRecord(name, null /* doc */, space, error);
+ schema = Schema.createRecord(name, doc, space, error);
consumeAvroAliasAnnotation(c, schema);
names.put(c.getName(), schema);
for (Field field : getCachedFields(c))
@@ -611,6 +613,8 @@ protected Schema createSchema(Type type, Map
names) {
JsonNode defaultValue = (defaultAnnotation == null)
? null
: Schema.parseJson(defaultAnnotation.value());
+ annotatedDoc = field.getAnnotation(AvroDoc.class);//
Docstring
+ doc = (annotatedDoc != null) ? annotatedDoc.value() : null;
if (defaultValue == null
&& fieldSchema.getType() == Schema.Type.UNION) {
@@ -624,7 +628,7 @@ protected Schema createSchema(Type type, Map
names) {
? annotatedName.value()
: field.getName();
Schema.Field recordField
-= new Schema.Field(fieldName, fieldSchema, null, defaultValue);
+= new Schema.Field(fieldName, fieldSchema, doc, defaultValue);
AvroMeta meta = field.getAnnotation(AvroMeta.class);
// add metadata
if (meta != null)
diff --git
a/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java
b/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java
index afe7f08ae..ddca2f9bb 100644
--- a/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java
+++ b/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java
@@
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693753#comment-16693753
]
ASF subversion and git services commented on AVRO-1658:
---
Commit 01a347e996a20406b0bf13026dd375725c4efb7f in avro's branch
refs/heads/master from [~aeroevan]
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=01a347e ]
AVRO-1658: Java: Add reflection annotation @AvroDoc.
@AvroMeta can be used for other keys, but doc is passed into the field's
constructor.
Patch 2: @AvroDoc should work for class-level documentation too.
Closes #189
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Zhaonan Sun
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[
https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693717#comment-16693717
]
ASF GitHub Bot commented on AVRO-1961:
--
dkulp closed pull request #169: Branch 1.8 - AVRO-1961 : [JAVA] Generate
getters that return an Optional
URL: https://github.com/apache/avro/pull/169
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/java/avro/src/main/java/org/apache/avro/Schema.java
b/lang/java/avro/src/main/java/org/apache/avro/Schema.java
index 2019c1f7d..7a962a09d 100644
--- a/lang/java/avro/src/main/java/org/apache/avro/Schema.java
+++ b/lang/java/avro/src/main/java/org/apache/avro/Schema.java
@@ -371,6 +371,26 @@ final boolean equalCachedHash(Schema other) {
"default","doc","name","order","type","aliases");
}
+ /** Returns true if this record is an union type. */
+ public boolean isUnion(){
+return this instanceof UnionSchema;
+ }
+
+ /** Returns true if this record is an union type containing null. */
+ public boolean isNullable() {
+if (!isUnion()) {
+ return getType().equals(Schema.Type.NULL);
+}
+
+for (Schema schema : getTypes()) {
+ if (schema.isNullable()) {
+return true;
+ }
+}
+
+return false;
+ }
+
/** A field within a record. */
public static class Field extends JsonProperties {
diff --git a/lang/java/avro/src/test/java/org/apache/avro/TestSchema.java
b/lang/java/avro/src/test/java/org/apache/avro/TestSchema.java
index 242ee8ca0..f75b0ed5f 100644
--- a/lang/java/avro/src/test/java/org/apache/avro/TestSchema.java
+++ b/lang/java/avro/src/test/java/org/apache/avro/TestSchema.java
@@ -18,11 +18,13 @@
package org.apache.avro;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.avro.Schema.Field;
@@ -33,35 +35,36 @@
@Test
public void testSplitSchemaBuild() {
Schema s = SchemaBuilder
- .record("HandshakeRequest")
- .namespace("org.apache.avro.ipc").fields()
- .name("clientProtocol").type().optional().stringType()
- .name("meta").type().optional().map().values().bytesType()
- .endRecord();
+.record("HandshakeRequest")
+.namespace("org.apache.avro.ipc").fields()
+.name("clientProtocol").type().optional().stringType()
+.name("meta").type().optional().map().values().bytesType()
+.endRecord();
String schemaString = s.toString();
-final int mid = schemaString.length() / 2;
+int mid = schemaString.length() / 2;
Schema parsedStringSchema = new
org.apache.avro.Schema.Parser().parse(s.toString());
Schema parsedArrayOfStringSchema =
- new org.apache.avro.Schema.Parser().parse
- (schemaString.substring(0, mid), schemaString.substring(mid));
+new org.apache.avro.Schema.Parser().parse
+(schemaString.substring(0, mid), schemaString.substring(mid));
assertNotNull(parsedStringSchema);
assertNotNull(parsedArrayOfStringSchema);
assertEquals(parsedStringSchema.toString(),
parsedArrayOfStringSchema.toString());
}
@Test
- public void testDuplicateRecordFieldName() {
-final Schema schema = Schema.createRecord("RecordName", null, null, false);
-final List fields = new ArrayList();
+ public void testDefaultRecordWithDuplicateFieldName() {
+String recordName = "name";
+Schema schema = Schema.createRecord(recordName, "doc", "namespace", false);
+List fields = new ArrayList();
fields.add(new Field("field_name", Schema.create(Type.NULL), null, null));
fields.add(new Field("field_name", Schema.create(Type.INT), null, null));
try {
schema.setFields(fields);
fail("Should not be able to create a record with duplicate field name.");
} catch (AvroRuntimeException are) {
- assertTrue(are.getMessage().contains("Duplicate field field_name in
record RecordName"));
+ assertTrue(are.getMessage().contains("Duplicate field field_name in
record " + recordName));
}
}
@@ -76,9 +79,23 @@ public void testCreateUnionVarargs() {
assertEquals(expected, schema);
}
+ @Test
+ public void testRecordWithNullDoc() {
+Schema schema = Schema.createRecord("name", null, "namespace", false);
+String schemaString = schema.toString();
+assertNotNull(schemaString);
+ }
+
+ @Test
+ public void testRecordWithNullNamespace() {
+Schema schema = Schema.createRecord("name", "doc", null,
[jira] [Resolved] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1961. --- Resolution: Fixed Fix Version/s: 1.9.0 > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Apache Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > Fix For: 1.9.0 > > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693716#comment-16693716 ] ASF subversion and git services commented on AVRO-1961: --- Commit 021c01c4c2756f793a4eee3a0f914c2c131823ac in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=021c01c ] Squashed commit of the following: commit a3f86c2874ad6d86b11fc2edc908065adcefdeda Author: Daniel Kulp Date: Tue Nov 20 14:32:48 2018 -0500 Grab some more tests from other PR commit 7d7822b5960c157ddf6db7fa15a3d797d0b286ee Author: Joseph Pachod Date: Sun Dec 11 22:50:45 2016 +0100 new javadoc with same formatting as other javadoc commit 82e8f7af6736034f4e85a4e6e1bcaa803019082e Author: Joseph Pachod Date: Sun Dec 11 22:38:33 2016 +0100 restore initial imports commit 0ca9815c8c884bc30c73b5c8a3c5d4c6e5188cee Author: Joseph Pachod Date: Sun Dec 11 22:36:03 2016 +0100 javadoc commit 551e1eb4a6bdedf00260ef90576490648f9b4658 Author: Joseph Pachod Date: Wed Dec 7 23:14:03 2016 +0100 format and clarify pre existing tests commit 857da0c70a2db321d5bf521b901fe0c035f32edc Author: Joseph Pachod Date: Wed Dec 7 22:53:13 2016 +0100 AVRO-1961: Java: add isUnion and isNullable on Schema class Closes #169 > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Apache Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > Fix For: 1.9.0 > > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2250) Release 1.9.0
[ https://issues.apache.org/jira/browse/AVRO-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693692#comment-16693692 ] Arianna Jakositz commented on AVRO-2250: Is there a ballpark time period for this new release? > Release 1.9.0 > - > > Key: AVRO-2250 > URL: https://issues.apache.org/jira/browse/AVRO-2250 > Project: Apache Avro > Issue Type: Task >Reporter: Nandor Kollar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1902) C: namespace "" should mean namespace = NULL
[
https://issues.apache.org/jira/browse/AVRO-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693684#comment-16693684
]
ASF GitHub Bot commented on AVRO-1902:
--
walshb edited a comment on issue #158: AVRO-1902: C: Namespace "" means space =
NULL.
URL: https://github.com/apache/avro/pull/158#issuecomment-440397330
I have no idea. All the people with commit access in Avro land don't seem to
understand C. I have 4 patches which they just ignore. I'm no longer using Avro
so I've just given up.
If you are interested in getting this fixed, I suggest finding the last
person who committed a significant change to the C code (ie. someone with
commit access, who cares about C), and contacting them directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C: namespace "" should mean namespace = NULL
>
>
> Key: AVRO-1902
> URL: https://issues.apache.org/jira/browse/AVRO-1902
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Reporter: Ben Walsh
>Priority: Minor
> Attachments: AVRO-1902.patch
>
>
> Similar to AVRO-1295 but for the C library. Namespace "" should be the NULL
> namespace when the schema is parsed. For example this breaks:
> ```
> {"type": "record", "name": "R", "fields": [
> {"name": "s", "type": {"type": "record", "namespace": "x", "name": "Y",
> "fields": [
> {"name": "e", "type": {"type": "record", "namespace": "", "name": "Z",
> "fields": [
> {"name": "f", "type": "Z"}
> ]}}
> ]}},
> {"name": "t", "type": "Z"}
> ]}
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1902) C: namespace "" should mean namespace = NULL
[
https://issues.apache.org/jira/browse/AVRO-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693682#comment-16693682
]
ASF GitHub Bot commented on AVRO-1902:
--
walshb commented on issue #158: AVRO-1902: C: Namespace "" means space = NULL.
URL: https://github.com/apache/avro/pull/158#issuecomment-440397330
I have no idea. All the people with commit access in Avro land don't seem to
understand C. I have 4 patches which the just ignore. I'm no longer using Avro
so I've just given up.
If you are interested in getting this fixed, I suggest finding the last
person who committed a significant change to the C code (ie. someone with
commit access, who cares about C), and contacting them directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C: namespace "" should mean namespace = NULL
>
>
> Key: AVRO-1902
> URL: https://issues.apache.org/jira/browse/AVRO-1902
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Reporter: Ben Walsh
>Priority: Minor
> Attachments: AVRO-1902.patch
>
>
> Similar to AVRO-1295 but for the C library. Namespace "" should be the NULL
> namespace when the schema is parsed. For example this breaks:
> ```
> {"type": "record", "name": "R", "fields": [
> {"name": "s", "type": {"type": "record", "namespace": "x", "name": "Y",
> "fields": [
> {"name": "e", "type": {"type": "record", "namespace": "", "name": "Z",
> "fields": [
> {"name": "f", "type": "Z"}
> ]}}
> ]}},
> {"name": "t", "type": "Z"}
> ]}
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693667#comment-16693667 ] ASF subversion and git services commented on AVRO-1961: --- Commit 04d4423d4ee4f86e308fa761c6918223b55a6c5c in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=04d4423 ] Squashed commit of the following: commit 6a919437bff0977926b33cd56165994b22fbdf74 Author: Niels Basjes Date: Fri Nov 25 10:31:40 2016 +0100 AVRO-1961: Extra flag to replace the regular getters with getters that return an Optional commit 91d58b5dba7562240a84f6885037f9c74dc973e6 Author: Niels Basjes Date: Tue Nov 22 17:23:18 2016 +0100 AVRO-1961: Java: Generate getters that return a Java 8 Optional. Closes #162 > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Apache Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[
https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693669#comment-16693669
]
ASF GitHub Bot commented on AVRO-1961:
--
dkulp closed pull request #162: AVRO-1961: Java: Generate getters that return a
Java 8 Optional.
URL: https://github.com/apache/avro/pull/162
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/CHANGES.txt b/CHANGES.txt
index 3553e2a93..e145d1cc1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -16,6 +16,8 @@ Trunk (not yet released)
AVRO-1932: Java: Allow setting the SchemaStore on generated classes.
(Niels Basjes)
+AVRO-1961: Java: Generate getters that return a Java 8 Optional. (Niels
Basjes)
+
OPTIMIZATIONS
IMPROVEMENTS
diff --git
a/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
b/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
index 295949313..36e51c9fc 100644
---
a/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
+++
b/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java
@@ -105,6 +105,8 @@
private VelocityEngine velocityEngine;
private String templateDir;
private FieldVisibility fieldVisibility = FieldVisibility.PUBLIC_DEPRECATED;
+ private boolean createOptionalGetters = false;
+ private boolean gettersReturnOptional = false;
private boolean createSetters = true;
private boolean createAllArgsConstructor = true;
private String outputCharacterEncoding;
@@ -217,6 +219,28 @@ public void setCreateSetters(boolean createSetters) {
this.createSetters = createSetters;
}
+ public boolean isCreateOptionalGetters() {
+return this.createOptionalGetters;
+ }
+
+ /**
+ * Set to false to not create the getters that return an Optional.
+ */
+ public void setCreateOptionalGetters(boolean createOptionalGetters) {
+this.createOptionalGetters = createOptionalGetters;
+ }
+
+ public boolean isGettersReturnOptional() {
+return this.gettersReturnOptional;
+ }
+
+ /**
+ * Set to false to not create the getters that return an Optional.
+ */
+ public void setGettersReturnOptional(boolean gettersReturnOptional) {
+this.gettersReturnOptional = gettersReturnOptional;
+ }
+
/**
* Set to true to use {@link java.math.BigDecimal} instead of
* {@link java.nio.ByteBuffer} for logical type "decimal"
@@ -778,6 +802,16 @@ public static String generateGetMethod(Schema schema,
Field field) {
return generateMethodName(schema, field, "get", "");
}
+ /**
+ * Generates the name of a field accessor method that returns a Java 8
Optional.
+ * @param schema the schema in which the field is defined.
+ * @param field the field for which to generate the accessor name.
+ * @return the name of the accessor method for the given field.
+ */
+ public static String generateGetOptionalMethod(Schema schema, Field field) {
+return generateMethodName(schema, field, "getOptional", "");
+ }
+
/**
* Generates the name of a field mutator method.
* @param schema the schema in which the field is defined.
diff --git
a/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
b/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
index 85c5e9d2f..04666e42a 100644
---
a/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
+++
b/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
@@ -25,6 +25,7 @@ import org.apache.avro.message.BinaryMessageEncoder;
import org.apache.avro.message.BinaryMessageDecoder;
import org.apache.avro.message.SchemaStore;
#end
+#if (${this.gettersReturnOptional} || ${this.createOptionalGetters})import
java.util.Optional;#end
@SuppressWarnings("all")
#if ($schema.getDoc())
@@ -184,6 +185,17 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
}
#foreach ($field in $schema.getFields())
+#if (${this.gettersReturnOptional})
+ /**
+ * Gets the value of the '${this.mangle($field.name(), $schema.isError())}'
field as an Optional<${this.javaType($field.schema())}>.
+#if ($field.doc()) * $field.doc()
+#end
+ * @return The value wrapped in an
Optional${this.javaType($field.schema())}.
+ */
+ public Optional<${this.javaType($field.schema())}>
${this.generateGetMethod($schema, $field)}() {
+return
Optional.<${this.javaType($field.schema())}>ofNullable(${this.mangle($field.name(),
$schema.isError())});
+ }
+#else
/**
*
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693668#comment-16693668 ] ASF subversion and git services commented on AVRO-1961: --- Commit 04d4423d4ee4f86e308fa761c6918223b55a6c5c in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=04d4423 ] Squashed commit of the following: commit 6a919437bff0977926b33cd56165994b22fbdf74 Author: Niels Basjes Date: Fri Nov 25 10:31:40 2016 +0100 AVRO-1961: Extra flag to replace the regular getters with getters that return an Optional commit 91d58b5dba7562240a84f6885037f9c74dc973e6 Author: Niels Basjes Date: Tue Nov 22 17:23:18 2016 +0100 AVRO-1961: Java: Generate getters that return a Java 8 Optional. Closes #162 > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Apache Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2244) Problems with TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
[
https://issues.apache.org/jira/browse/AVRO-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693051#comment-16693051
]
ASF subversion and git services commented on AVRO-2244:
---
Commit e04c2c291395339b8eb488a5e045500d366553b6 in avro's branch
refs/heads/master from Thiruvalluvan M G
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=e04c2c2 ]
AVRO-2244 Fix flaky datetime test (#386)
> Problems with
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
> ---
>
> Key: AVRO-2244
> URL: https://issues.apache.org/jira/browse/AVRO-2244
> Project: Apache Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Thiruvalluvan M. G.
>Priority: Major
> Fix For: 1.9.0
>
>
> I've seen an intermittent test failure that looks like this:
> {{Failed tests:}}
> {{
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148}}
> {{Expected: is "20:35:18.720"}}
> {{ but: was "20:35:18.72"}}
> When I see this failure, it's always the case that the trailing digit is
> zero. I suspect that it's a bug where the trailing zero is not printed.
> Since the test cases use the current time, then most of the time the trailing
> digit isn't zero and the bug isn't tickled. But once-in-a-while the current
> time has a trailing zero, which tickles the bug.
> If this diagnosis is correct, then in addition to fixing the bug, it might be
> a good idea to add tests with hard-wired, static times that cover corner
> cases like this one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2244) Problems with TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
[
https://issues.apache.org/jira/browse/AVRO-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AVRO-2244.
Resolution: Fixed
Fix Version/s: 1.9.0
> Problems with
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
> ---
>
> Key: AVRO-2244
> URL: https://issues.apache.org/jira/browse/AVRO-2244
> Project: Apache Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Thiruvalluvan M. G.
>Priority: Major
> Fix For: 1.9.0
>
>
> I've seen an intermittent test failure that looks like this:
> {{Failed tests:}}
> {{
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148}}
> {{Expected: is "20:35:18.720"}}
> {{ but: was "20:35:18.72"}}
> When I see this failure, it's always the case that the trailing digit is
> zero. I suspect that it's a bug where the trailing zero is not printed.
> Since the test cases use the current time, then most of the time the trailing
> digit isn't zero and the bug isn't tickled. But once-in-a-while the current
> time has a trailing zero, which tickles the bug.
> If this diagnosis is correct, then in addition to fixing the bug, it might be
> a good idea to add tests with hard-wired, static times that cover corner
> cases like this one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2244) Problems with TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
[
https://issues.apache.org/jira/browse/AVRO-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693047#comment-16693047
]
ASF GitHub Bot commented on AVRO-2244:
--
Fokko commented on issue #386: Test and fix for AVRO-2244
URL: https://github.com/apache/avro/pull/386#issuecomment-440232578
This looks good. Thanks @thiru-apache for fixing this!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Problems with
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
> ---
>
> Key: AVRO-2244
> URL: https://issues.apache.org/jira/browse/AVRO-2244
> Project: Apache Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Thiruvalluvan M. G.
>Priority: Major
> Fix For: 1.9.0
>
>
> I've seen an intermittent test failure that looks like this:
> {{Failed tests:}}
> {{
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148}}
> {{Expected: is "20:35:18.720"}}
> {{ but: was "20:35:18.72"}}
> When I see this failure, it's always the case that the trailing digit is
> zero. I suspect that it's a bug where the trailing zero is not printed.
> Since the test cases use the current time, then most of the time the trailing
> digit isn't zero and the bug isn't tickled. But once-in-a-while the current
> time has a trailing zero, which tickles the bug.
> If this diagnosis is correct, then in addition to fixing the bug, it might be
> a good idea to add tests with hard-wired, static times that cover corner
> cases like this one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2244) Problems with TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
[
https://issues.apache.org/jira/browse/AVRO-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693050#comment-16693050
]
ASF GitHub Bot commented on AVRO-2244:
--
Fokko closed pull request #386: Test and fix for AVRO-2244
URL: https://github.com/apache/avro/pull/386
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/lang/java/avro/src/test/java/org/apache/avro/specific/TestSpecificLogicalTypes.java
b/lang/java/avro/src/test/java/org/apache/avro/specific/TestSpecificLogicalTypes.java
index ba8923980..02bea73bd 100644
---
a/lang/java/avro/src/test/java/org/apache/avro/specific/TestSpecificLogicalTypes.java
+++
b/lang/java/avro/src/test/java/org/apache/avro/specific/TestSpecificLogicalTypes.java
@@ -27,7 +27,6 @@
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
import java.time.temporal.ChronoField;
-import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
@@ -46,6 +45,7 @@
import org.joda.time.DateTimeZone;
import org.joda.time.LocalDate;
import org.joda.time.LocalTime;
+import org.joda.time.chrono.ISOChronology;
import org.joda.time.format.ISODateTimeFormat;
import org.junit.Assert;
import org.junit.Rule;
@@ -119,8 +119,8 @@ public void testRecordWithJsr310LogicalTypes() throws
IOException {
3019.34,
null,
java.time.LocalDate.now(),
-java.time.LocalTime.now().truncatedTo(ChronoUnit.MILLIS),
-java.time.Instant.now().truncatedTo(ChronoUnit.MILLIS),
+java.time.LocalTime.now(),
+java.time.Instant.now(),
new BigDecimal(123.45f).setScale(2, BigDecimal.ROUND_HALF_DOWN)
);
@@ -133,6 +133,7 @@ public void testRecordWithJsr310LogicalTypes() throws
IOException {
@Test
public void testAbilityToReadJsr310RecordWrittenAsJodaRecord() throws
IOException {
+
TestRecordWithLogicalTypes withJoda = new TestRecordWithLogicalTypes(
true,
34,
@@ -142,7 +143,10 @@ public void
testAbilityToReadJsr310RecordWrittenAsJodaRecord() throws IOExceptio
null,
LocalDate.now(),
LocalTime.now(),
-DateTime.now().withZone(DateTimeZone.UTC),
+// There is no reliable way to get fixed width string from
ISO_INSTANT below
+// for granularity less than one second second.
+new DateTime((System.currentTimeMillis() / 1000) * 1000,
+ISOChronology.getInstance()).withZone(DateTimeZone.UTC),
new BigDecimal(123.45f).setScale(2, BigDecimal.ROUND_HALF_DOWN)
);
@@ -162,7 +166,7 @@ public void
testAbilityToReadJsr310RecordWrittenAsJodaRecord() throws IOExceptio
Assert.assertThat(ISO_LOCAL_DATE.format(withJsr310.getD()),
is(ISODateTimeFormat.date().print(withJoda.getD(;
Assert.assertThat(ISO_LOCAL_TIME.format(withJsr310.getT()),
is(ISODateTimeFormat.time().print(withJoda.getT(;
-Assert.assertThat(ISO_INSTANT.format(withJsr310.getTs()),
is(ISODateTimeFormat.dateTime().print(withJoda.getTs(;
+Assert.assertThat(ISO_INSTANT.format(withJsr310.getTs()),
is(ISODateTimeFormat.dateTimeNoMillis().print(withJoda.getTs(;
Assert.assertThat(withJsr310.getDec(), comparesEqualTo(withJoda.getDec()));
}
@@ -176,8 +180,8 @@ public void
testAbilityToReadJodaRecordWrittenAsJsr310Record() throws IOExceptio
3019.34,
null,
java.time.LocalDate.now(),
-java.time.LocalTime.now().truncatedTo(ChronoUnit.MILLIS),
-java.time.Instant.now().truncatedTo(ChronoUnit.MILLIS),
+java.time.LocalTime.now(),
+java.time.Instant.now(),
new BigDecimal(123.45f).setScale(2, BigDecimal.ROUND_HALF_DOWN)
);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Problems with
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
> ---
>
> Key: AVRO-2244
> URL: https://issues.apache.org/jira/browse/AVRO-2244
> Project: Apache Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Thiruvalluvan M. G.
>Priority: Major
> Fix For: 1.9.0
>
>
> I've seen an intermittent test failure that looks like this:
>
[jira] [Resolved] (AVRO-2241) Fix Jsr310/Joda tests when comparing datetimes
[
https://issues.apache.org/jira/browse/AVRO-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AVRO-2241.
Resolution: Fixed
> Fix Jsr310/Joda tests when comparing datetimes
> --
>
> Key: AVRO-2241
> URL: https://issues.apache.org/jira/browse/AVRO-2241
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.9.0
>
>
> Whe comparing Instant, DateTime, LocalTime with its Joda equivalents the
> tests fail since the java.time.* adds support for microseconds, and Joda only
> miliseconds. For example:
> Failed tests:
>
> TestSpecificLogicalTypes.testAbilityToReadJodaRecordWrittenAsJsr310Record:183
> Expected: is "11:45:07.170331"
> but: was "11:45:07.170"
> TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes:115 Should match
> written record expected:<{"b": true, "i32": 34, "i64": 35, "f32": 3.14,
> "f64": 3019.34, "s": null, "d": 2018-10-11, "t": 11:45:06.513252, "ts":
> 2018-10-11T11:45:06.513358Z, "dec": 123.45}> but was:<{"b": true, "i32": 34,
> "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018-10-11, "t":
> 11:45:06.513, "ts": 2018-10-11T11:45:06.513Z, "dec": 123.45}>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2244) Problems with TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
[
https://issues.apache.org/jira/browse/AVRO-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693045#comment-16693045
]
Fokko Driesprong commented on AVRO-2244:
I fixed this over here:
https://github.com/apache/avro/commit/25032e10a86851bdabfa29b99885ccfc03c6667e#diff-a2bcc0c4e92dfb4332f75a9d56cdb4daR73
> Problems with
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
> ---
>
> Key: AVRO-2244
> URL: https://issues.apache.org/jira/browse/AVRO-2244
> Project: Apache Avro
> Issue Type: Bug
> Components: logical types
>Reporter: Raymie Stata
>Assignee: Thiruvalluvan M. G.
>Priority: Major
>
> I've seen an intermittent test failure that looks like this:
> {{Failed tests:}}
> {{
> TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148}}
> {{Expected: is "20:35:18.720"}}
> {{ but: was "20:35:18.72"}}
> When I see this failure, it's always the case that the trailing digit is
> zero. I suspect that it's a bug where the trailing zero is not printed.
> Since the test cases use the current time, then most of the time the trailing
> digit isn't zero and the bug isn't tickled. But once-in-a-while the current
> time has a trailing zero, which tickles the bug.
> If this diagnosis is correct, then in addition to fixing the bug, it might be
> a good idea to add tests with hard-wired, static times that cover corner
> cases like this one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2267) Duplicate code RandomData.java and its dependency problem
[
https://issues.apache.org/jira/browse/AVRO-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692922#comment-16692922
]
ASF subversion and git services commented on AVRO-2267:
---
Commit eb06f71861025c9261b5ad7a09cab0c46f2ac311 in avro's branch
refs/heads/master from Thiruvalluvan M G
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=eb06f71 ]
AVRO-2267 Removed copies of RandomData (#385)
AVRO-2267 Removed copies of RandomData
> Duplicate code RandomData.java and its dependency problem
> -
>
> Key: AVRO-2267
> URL: https://issues.apache.org/jira/browse/AVRO-2267
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Thiruvalluvan M. G.
>Assignee: Thiruvalluvan M. G.
>Priority: Major
>
> There are two issues with {{RandomData}} class:
> * There are almost identical copies of the same code in two modules:
> {{avro}} and {{avro-ipc}}. We should use a single source file.
> * Both the copies belong to {{test}} subfolders in their respective module.
> But {{avro-tools}} module uses this class in {{main}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2039) Ruby encoding performance improvement
[ https://issues.apache.org/jira/browse/AVRO-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AVRO-2039: --- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Ruby encoding performance improvement > - > > Key: AVRO-2039 > URL: https://issues.apache.org/jira/browse/AVRO-2039 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Critical > Fix For: 1.9.0 > > > For a use case with a few levels of nesting and unions in several places > within the schema we saw a 5x improvement in encoding performance with these > changes to encoding using Ruby. > 1. Avoid the exhaustive validation of schemas in a union > 2. Avoid the repeated validation of nested schemas > https://github.com/apache/avro/pull/230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2039) Ruby encoding performance improvement
[ https://issues.apache.org/jira/browse/AVRO-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692932#comment-16692932 ] ASF subversion and git services commented on AVRO-2039: --- Commit 39500088491512aeccf3e9dc7b76d99b91a82436 in avro's branch refs/heads/master from [~timperkins] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=3950008 ] AVRO-2039: Ruby encoding performance improvements (#230) > Ruby encoding performance improvement > - > > Key: AVRO-2039 > URL: https://issues.apache.org/jira/browse/AVRO-2039 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Critical > > For a use case with a few levels of nesting and unions in several places > within the schema we saw a 5x improvement in encoding performance with these > changes to encoding using Ruby. > 1. Avoid the exhaustive validation of schemas in a union > 2. Avoid the repeated validation of nested schemas > https://github.com/apache/avro/pull/230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2039) Ruby encoding performance improvement
[
https://issues.apache.org/jira/browse/AVRO-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692931#comment-16692931
]
ASF GitHub Bot commented on AVRO-2039:
--
Fokko closed pull request #230: AVRO-2039: Ruby encoding performance
improvements
URL: https://github.com/apache/avro/pull/230
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/ruby/lib/avro/io.rb b/lang/ruby/lib/avro/io.rb
index 26bda973a..958159c79 100644
--- a/lang/ruby/lib/avro/io.rb
+++ b/lang/ruby/lib/avro/io.rb
@@ -5,9 +5,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -45,7 +45,7 @@ def initialize(reader)
def byte!
@reader.read(1).unpack('C').first
end
-
+
def read_null
# null is written as zero byte's
nil
@@ -159,7 +159,7 @@ def write_null(datum)
nil
end
- # a boolean is written as a single byte
+ # a boolean is written as a single byte
# whose value is either 0 (false) or 1 (true).
def write_boolean(datum)
on_disk = datum ? 1.chr : 0.chr
@@ -504,7 +504,7 @@ def write(datum, encoder)
def write_data(writers_schema, logical_datum, encoder)
datum = writers_schema.type_adapter.encode(logical_datum)
-unless Schema.validate(writers_schema, datum, encoded = true)
+unless Schema.validate(writers_schema, datum, { recursive: false,
encoded: true })
raise AvroTypeError.new(writers_schema, datum)
end
@@ -539,6 +539,7 @@ def write_enum(writers_schema, datum, encoder)
end
def write_array(writers_schema, datum, encoder)
+raise AvroTypeError.new(writers_schema, datum) unless
datum.is_a?(Array)
if datum.size > 0
encoder.write_long(datum.size)
datum.each do |item|
@@ -549,6 +550,7 @@ def write_array(writers_schema, datum, encoder)
end
def write_map(writers_schema, datum, encoder)
+raise AvroTypeError.new(writers_schema, datum) unless datum.is_a?(Hash)
if datum.size > 0
encoder.write_long(datum.size)
datum.each do |k,v|
@@ -571,6 +573,7 @@ def write_union(writers_schema, datum, encoder)
end
def write_record(writers_schema, datum, encoder)
+raise AvroTypeError.new(writers_schema, datum) unless datum.is_a?(Hash)
writers_schema.fields.each do |field|
write_data(field.type, datum[field.name], encoder)
end
diff --git a/lang/ruby/lib/avro/schema.rb b/lang/ruby/lib/avro/schema.rb
index 3acd07b7f..c1b150ead 100644
--- a/lang/ruby/lib/avro/schema.rb
+++ b/lang/ruby/lib/avro/schema.rb
@@ -96,8 +96,8 @@ def self.real_parse(json_obj, names=nil,
default_namespace=nil)
end
# Determine if a ruby datum is an instance of a schema
-def self.validate(expected_schema, logical_datum, encoded = false)
- SchemaValidator.validate!(expected_schema, logical_datum, encoded)
+def self.validate(expected_schema, logical_datum, options = { recursive:
true, encoded: false })
+ SchemaValidator.validate!(expected_schema, logical_datum, options)
true
rescue SchemaValidator::ValidationError
false
diff --git a/lang/ruby/lib/avro/schema_validator.rb
b/lang/ruby/lib/avro/schema_validator.rb
index 2117e92d4..28022a8c6 100644
--- a/lang/ruby/lib/avro/schema_validator.rb
+++ b/lang/ruby/lib/avro/schema_validator.rb
@@ -20,7 +20,8 @@ class SchemaValidator
PATH_SEPARATOR = '.'.freeze
INT_RANGE = Schema::INT_MIN_VALUE..Schema::INT_MAX_VALUE
LONG_RANGE = Schema::LONG_MIN_VALUE..Schema::LONG_MAX_VALUE
-COMPLEX_TYPES = [:array, :error, :map, :record, :request]
+COMPLEX_TYPES = [:array, :error, :map, :record, :request].freeze
+BOOLEAN_VALUES = [true, false].freeze
class Result
attr_reader :errors
@@ -62,27 +63,54 @@ def to_s
TypeMismatchError = Class.new(ValidationError)
class << self
- def validate!(expected_schema, logical_datum, encoded = false)
+ def validate!(expected_schema, logical_datum, options = { recursive:
true, encoded: false })
+options ||= {}
+options[:recursive] = true unless options.key?(:recursive)
+
result = Result.new
-validate_recursive(expected_schema, logical_datum, ROOT_IDENTIFIER,
result,
[jira] [Commented] (AVRO-2267) Duplicate code RandomData.java and its dependency problem
[
https://issues.apache.org/jira/browse/AVRO-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692923#comment-16692923
]
ASF subversion and git services commented on AVRO-2267:
---
Commit eb06f71861025c9261b5ad7a09cab0c46f2ac311 in avro's branch
refs/heads/master from Thiruvalluvan M G
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=eb06f71 ]
AVRO-2267 Removed copies of RandomData (#385)
AVRO-2267 Removed copies of RandomData
> Duplicate code RandomData.java and its dependency problem
> -
>
> Key: AVRO-2267
> URL: https://issues.apache.org/jira/browse/AVRO-2267
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Thiruvalluvan M. G.
>Assignee: Thiruvalluvan M. G.
>Priority: Major
>
> There are two issues with {{RandomData}} class:
> * There are almost identical copies of the same code in two modules:
> {{avro}} and {{avro-ipc}}. We should use a single source file.
> * Both the copies belong to {{test}} subfolders in their respective module.
> But {{avro-tools}} module uses this class in {{main}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2267) Duplicate code RandomData.java and its dependency problem
[
https://issues.apache.org/jira/browse/AVRO-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692921#comment-16692921
]
ASF GitHub Bot commented on AVRO-2267:
--
Fokko closed pull request #385: AVRO-2267 Removed copies of RandomData
URL: https://github.com/apache/avro/pull/385
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/java/avro/pom.xml b/lang/java/avro/pom.xml
index c9104fba9..a684b1ccc 100644
--- a/lang/java/avro/pom.xml
+++ b/lang/java/avro/pom.xml
@@ -106,8 +106,7 @@
interop-generate-null-codec
generate-resources
- org.apache.avro.RandomData
- test
+ org.apache.avro.util.RandomData
../../../share/test/schemas/interop.avsc
../../../build/interop/data/java.avro
@@ -121,8 +120,7 @@
interop-generate-deflate-codec
generate-resources
- org.apache.avro.RandomData
- test
+ org.apache.avro.util.RandomData
../../../share/test/schemas/interop.avsc
../../../build/interop/data/java_deflate.avro
diff --git a/lang/java/avro/src/test/java/org/apache/avro/RandomData.java
b/lang/java/avro/src/main/java/org/apache/avro/util/RandomData.java
similarity index 72%
rename from lang/java/avro/src/test/java/org/apache/avro/RandomData.java
rename to lang/java/avro/src/main/java/org/apache/avro/util/RandomData.java
index 89a4321d6..b8c7c042d 100644
--- a/lang/java/avro/src/test/java/org/apache/avro/RandomData.java
+++ b/lang/java/avro/src/main/java/org/apache/avro/util/RandomData.java
@@ -15,34 +15,51 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.avro;
+package org.apache.avro.util;
import java.io.File;
import java.nio.ByteBuffer;
-import java.util.*;
+import java.nio.charset.Charset;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import org.apache.avro.Schema;
import org.apache.avro.file.CodecFactory;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericArray;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
-import org.apache.avro.util.Utf8;
/** Generates schema data as Java objects with random values. */
public class RandomData implements Iterable {
+ public static final String USE_DEFAULT = "use-default";
+
private final Schema root;
private final long seed;
private final int count;
+ private final boolean utf8ForString;
public RandomData(Schema schema, int count) {
-this(schema, count, System.currentTimeMillis());
+this(schema, count, false);
}
public RandomData(Schema schema, int count, long seed) {
+this(schema, count, seed, false);
+ }
+
+ public RandomData(Schema schema, int count, boolean utf8ForString) {
+this(schema, count, System.currentTimeMillis(), utf8ForString);
+ }
+
+ public RandomData(Schema schema, int count, long seed, boolean
utf8ForString) {
this.root = schema;
this.seed = seed;
this.count = count;
+this.utf8ForString = utf8ForString;
}
public Iterator iterator() {
@@ -59,12 +76,16 @@ public Object next() {
}
@SuppressWarnings(value="unchecked")
- private static Object generate(Schema schema, Random random, int d) {
+ private Object generate(Schema schema, Random random, int d) {
switch (schema.getType()) {
case RECORD:
GenericRecord record = new GenericData.Record(schema);
- for (Schema.Field field : schema.getFields())
-record.put(field.name(), generate(field.schema(), random, d+1));
+ for (Schema.Field field : schema.getFields()) {
+Object value = (field.getObjectProp(USE_DEFAULT) == null) ?
+generate(field.schema(), random, d+1) :
+GenericData.get().getDefaultValue(field);
+record.put(field.name(), value);
+ }
return record;
case ENUM:
List symbols = schema.getEnumSymbols();
@@ -72,6 +93,7 @@ private static Object generate(Schema schema, Random random,
int d) {
(schema, symbols.get(random.nextInt(symbols.size(;
case ARRAY:
int length = (random.nextInt(5)+2)-d;
+ @SuppressWarnings("rawtypes")
GenericArray array =
new GenericData.Array(length<=0?0:length, schema);
for (int i = 0; i < length; i++)
@@
[jira] [Resolved] (AVRO-2267) Duplicate code RandomData.java and its dependency problem
[
https://issues.apache.org/jira/browse/AVRO-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AVRO-2267.
Resolution: Fixed
Fix Version/s: 1.9.0
> Duplicate code RandomData.java and its dependency problem
> -
>
> Key: AVRO-2267
> URL: https://issues.apache.org/jira/browse/AVRO-2267
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Thiruvalluvan M. G.
>Assignee: Thiruvalluvan M. G.
>Priority: Major
> Fix For: 1.9.0
>
>
> There are two issues with {{RandomData}} class:
> * There are almost identical copies of the same code in two modules:
> {{avro}} and {{avro-ipc}}. We should use a single source file.
> * Both the copies belong to {{test}} subfolders in their respective module.
> But {{avro-tools}} module uses this class in {{main}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2271) C++ Support for Custom Fields
[
https://issues.apache.org/jira/browse/AVRO-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692902#comment-16692902
]
ASF GitHub Bot commented on AVRO-2271:
--
Fokko commented on issue #387: AVRO-2271: C++ Support for Custom Fields
URL: https://github.com/apache/avro/pull/387#issuecomment-440199939
Sad CI:
```
Scanning dependencies of target avrocpp_s
[ 1%] Building CXX object CMakeFiles/avrocpp_s.dir/impl/Compiler.cc.o
/testptch/unknown/lang/c++/impl/Compiler.cc: In function ‘void
avro::addCustomFields(const NodePtr&, const Object&)’:
/testptch/unknown/lang/c++/impl/Compiler.cc:299:53: error: in C++98
‘kKnownFields’ must be initialized by constructor, not by ‘{...}’
"values", "precision", "scale", "namespace"};
^
/testptch/unknown/lang/c++/impl/Compiler.cc:299:53: error: could not convert
‘{"name", "type", "default", "doc", "size", "logicalType", "values",
"precision", "scale", "namespace"}’ from ‘’ to ‘const std::__debug::vector,
std::allocator > >’
/testptch/unknown/lang/c++/impl/Compiler.cc:300:16: warning: ‘auto’
changes meaning in C++11; please remove it [-Wc++0x-compat]
for (const auto : m) {
^
/testptch/unknown/lang/c++/impl/Compiler.cc:300:22: error: ISO C++ forbids
declaration of ‘entry’ with no type [-fpermissive]
for (const auto : m) {
^
/testptch/unknown/lang/c++/impl/Compiler.cc:300:30: error: range-based
‘for’ loops are not allowed in C++98 mode
for (const auto : m) {
^
/testptch/unknown/lang/c++/impl/Compiler.cc:301:71: error: request for
member ‘first’ in ‘entry’, which is of non-class type ‘const int’
if (std::find(kKnownFields.begin(), kKnownFields.end(), entry.first)
^
/testptch/unknown/lang/c++/impl/Compiler.cc:303:40: error: request for
member ‘first’ in ‘entry’, which is of non-class type ‘const int’
node->addCustomField(entry.first, entry.second);
^
/testptch/unknown/lang/c++/impl/Compiler.cc:303:53: error: request for
member ‘second’ in ‘entry’, which is of non-class type ‘const int’
node->addCustomField(entry.first, entry.second);
^
make[2]: *** [CMakeFiles/avrocpp_s.dir/impl/Compiler.cc.o] Error 1
CMakeFiles/avrocpp_s.dir/build.make:54: recipe for target
'CMakeFiles/avrocpp_s.dir/impl/Compiler.cc.o' failed
make[1]: *** [CMakeFiles/avrocpp_s.dir/all] Error 2
CMakeFiles/Makefile2:425: recipe for target 'CMakeFiles/avrocpp_s.dir/all'
failed
make: *** [all] Error 2
Makefile:147: recipe for target 'all' failed
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C++ Support for Custom Fields
> --
>
> Key: AVRO-2271
> URL: https://issues.apache.org/jira/browse/AVRO-2271
> Project: Apache Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.2
>Reporter: Aniket Mokashi
>Assignee: Aniket Mokashi
>Priority: Major
>
> To support following schemas:
>
> {noformat}
> R"({"type": "string", "logicalType": "foo", "sqlType": "datetime"})",
> R"({"type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 6,
> "sqlType": "datetime"})",
> R"({"type": "null", "extra attribute": "custom field"})",
> R"({"type": "boolean", "extra1": 1, "extra2": 2, "extra3": 3})",
> R"({"type": "fixed", "name": "Test", "size": 1, "extra attribute": 1})",
> R"(["null", \{"type": "string", "extra attribute": "custom field"}])",
> R"([\{"type": "string", "extra attribute": "custom field"}, "null"])",
> {noformat}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
