[jira] [Commented] (AVRO-1667) Parser symbol tree flattening is broken for recursive schemas
[
https://issues.apache.org/jira/browse/AVRO-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610764#comment-14610764
]
Zoltan Farkas commented on AVRO-1667:
-
A tough one, it was tuff to fix this more than a month ago...
when copying the production p with:
System.arraycopy(p, 0, out, j, p.length);
if case there is a Fixup for this production, (the production will be
constructed later), this array copy will copy nulls... (this is the source of
the nulls)
what the patch does by adding after the array copy:
for (ListFixup value : map2.values()) {
for (Fixup fixup : value) {
if (fixup.symbols == p) {
fixup.symbols = out;
fixup.pos += j;
}
}
}
Update all the Fixups for P to the destination of the array copy
I hope I make sense, let me know otherwise...
Parser symbol tree flattening is broken for recursive schemas
-
Key: AVRO-1667
URL: https://issues.apache.org/jira/browse/AVRO-1667
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.7
Reporter: Zoltan Farkas
Here is a unit test to reproduce:
{noformat}
package org.apache.avro.io.parsing;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Set;
import junit.framework.Assert;
import org.apache.avro.Schema;
import org.junit.Test;
public class SymbolTest {
private static final String SCHEMA =
{\type\:\record\,\name\:\SampleNode\,
+ \namespace\:\org.spf4j.ssdump2.avro\,\n +
\fields\:[\n +
{\name\:\count\,\type\:\int\,\default\:0},\n +
{\name\:\subNodes\,\type\:\n +
{\type\:\array\,\items\:{\n +
\type\:\record\,\name\:\SamplePair\,\n +
\fields\:[\n +
{\name\:\method\,\type\:\n +
{\type\:\record\,\name\:\Method\,\n +
\fields\:[\n +
{\name\:\declaringClass\,\type\:{\type\:\string\,\avro.java.string\:\String\}},\n
+
{\name\:\methodName\,\type\:{\type\:\string\,\avro.java.string\:\String\}}\n
+
]}},\n +
{\name\:\node\,\type\:\SampleNode\}]}}}]};
@Test
public void testSomeMethod() throws IOException {
Schema schema = new Schema.Parser().parse(SCHEMA);
Symbol root = Symbol.root(new ResolvingGrammarGenerator()
.generate(schema, schema, new
HashMapValidatingGrammarGenerator.LitS, Symbol()));
validateNonNull(root, new HashSetSymbol());
}
private static void validateNonNull(final Symbol symb, SetSymbol seen) {
if (seen.contains(symb)) {
return;
} else {
seen.add(symb);
}
if (symb.production != null) {
for (Symbol s : symb.production) {
if (s == null) {
Assert.fail(invalid parsing tree should not contain
nulls);
}
if (s.kind != Symbol.Kind.ROOT) {
validateNonNull(s, seen);;
}
}
}
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1667) Parser symbol tree flattening is broken for recursive schemas
[
https://issues.apache.org/jira/browse/AVRO-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610769#comment-14610769
]
Ryan Blue commented on AVRO-1667:
-
I think I see what you're saying: when flattening a recursive structure, it
makes a copy and needs to fill in values later (from the Fixup, when it is
available) but wasn't doing that. Does that sound correct?
Parser symbol tree flattening is broken for recursive schemas
-
Key: AVRO-1667
URL: https://issues.apache.org/jira/browse/AVRO-1667
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.7
Reporter: Zoltan Farkas
Here is a unit test to reproduce:
{noformat}
package org.apache.avro.io.parsing;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Set;
import junit.framework.Assert;
import org.apache.avro.Schema;
import org.junit.Test;
public class SymbolTest {
private static final String SCHEMA =
{\type\:\record\,\name\:\SampleNode\,
+ \namespace\:\org.spf4j.ssdump2.avro\,\n +
\fields\:[\n +
{\name\:\count\,\type\:\int\,\default\:0},\n +
{\name\:\subNodes\,\type\:\n +
{\type\:\array\,\items\:{\n +
\type\:\record\,\name\:\SamplePair\,\n +
\fields\:[\n +
{\name\:\method\,\type\:\n +
{\type\:\record\,\name\:\Method\,\n +
\fields\:[\n +
{\name\:\declaringClass\,\type\:{\type\:\string\,\avro.java.string\:\String\}},\n
+
{\name\:\methodName\,\type\:{\type\:\string\,\avro.java.string\:\String\}}\n
+
]}},\n +
{\name\:\node\,\type\:\SampleNode\}]}}}]};
@Test
public void testSomeMethod() throws IOException {
Schema schema = new Schema.Parser().parse(SCHEMA);
Symbol root = Symbol.root(new ResolvingGrammarGenerator()
.generate(schema, schema, new
HashMapValidatingGrammarGenerator.LitS, Symbol()));
validateNonNull(root, new HashSetSymbol());
}
private static void validateNonNull(final Symbol symb, SetSymbol seen) {
if (seen.contains(symb)) {
return;
} else {
seen.add(symb);
}
if (symb.production != null) {
for (Symbol s : symb.production) {
if (s == null) {
Assert.fail(invalid parsing tree should not contain
nulls);
}
if (s.kind != Symbol.Kind.ROOT) {
validateNonNull(s, seen);;
}
}
}
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Next Avro release
+1 for a 1.8.0 release. What kind of timeline for the first RC are you looking at? On Wed, Jul 1, 2015 at 10:00 AM, Zoltan Farkas zolyfar...@yahoo.com.invalid wrote: I would like to have https://issues.apache.org/jira/browse/AVRO-1667 https://issues.apache.org/jira/browse/AVRO-1667 resolved in 1.8.0 if possible. This issue prevents serializing/deserializing objects with certain schemas… —Z On Jul 1, 2015, at 10:50 AM, Niels Basjes ni...@basjes.nl wrote: Hi, I would say go for the 1.8.0 Note that there are a few issues that seem appropriate to include for this one because they relate to a change in 'backwards compatibility: - AVRO-1586 Build against Hadoop 2 ( which should also fix AVRO-1453 Release version of avro-tools compiled against hadoop2 ) - AVRO-1559 Drop support for Ruby 1.8 On a personal note I would like a 'Yes, commit' / 'No, won't fix' choice from you guys regarding this proposal (Patch included): AVRO-1633 Add additional setXxx(Builder) method to make user code more readable. Niels Basjes On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com wrote: Hi everyone, It would be good to do another Avro release soon. I'm happy to create a release candidate so we can vote on it. What do folks think about releasing 1.7.8 vs. 1.8.0? There are very few changes in the 1.7 branch, so it might be best to release 1.8.0 from trunk. There have been enough changes to justify a new minor release I think. Are there any drawbacks to doing that? The unresolved 1.8.0 issues are here: http://s.apache.org/Cdt And the unresolved 1.7.8 issues are here: http://s.apache.org/5ShO Thanks, Tom -- Best regards / Met vriendelijke groeten, Niels Basjes -- Sean
Next Avro release
Hi everyone, It would be good to do another Avro release soon. I'm happy to create a release candidate so we can vote on it. What do folks think about releasing 1.7.8 vs. 1.8.0? There are very few changes in the 1.7 branch, so it might be best to release 1.8.0 from trunk. There have been enough changes to justify a new minor release I think. Are there any drawbacks to doing that? The unresolved 1.8.0 issues are here: http://s.apache.org/Cdt And the unresolved 1.7.8 issues are here: http://s.apache.org/5ShO Thanks, Tom
Re: Next Avro release
Hi, I would say go for the 1.8.0 Note that there are a few issues that seem appropriate to include for this one because they relate to a change in 'backwards compatibility: - AVRO-1586 Build against Hadoop 2 ( which should also fix AVRO-1453 Release version of avro-tools compiled against hadoop2 ) - AVRO-1559 Drop support for Ruby 1.8 On a personal note I would like a 'Yes, commit' / 'No, won't fix' choice from you guys regarding this proposal (Patch included): AVRO-1633 Add additional setXxx(Builder) method to make user code more readable. Niels Basjes On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com wrote: Hi everyone, It would be good to do another Avro release soon. I'm happy to create a release candidate so we can vote on it. What do folks think about releasing 1.7.8 vs. 1.8.0? There are very few changes in the 1.7 branch, so it might be best to release 1.8.0 from trunk. There have been enough changes to justify a new minor release I think. Are there any drawbacks to doing that? The unresolved 1.8.0 issues are here: http://s.apache.org/Cdt And the unresolved 1.7.8 issues are here: http://s.apache.org/5ShO Thanks, Tom -- Best regards / Met vriendelijke groeten, Niels Basjes
Re: Next Avro release
I would like to have https://issues.apache.org/jira/browse/AVRO-1667 https://issues.apache.org/jira/browse/AVRO-1667 resolved in 1.8.0 if possible. This issue prevents serializing/deserializing objects with certain schemas… —Z On Jul 1, 2015, at 10:50 AM, Niels Basjes ni...@basjes.nl wrote: Hi, I would say go for the 1.8.0 Note that there are a few issues that seem appropriate to include for this one because they relate to a change in 'backwards compatibility: - AVRO-1586 Build against Hadoop 2 ( which should also fix AVRO-1453 Release version of avro-tools compiled against hadoop2 ) - AVRO-1559 Drop support for Ruby 1.8 On a personal note I would like a 'Yes, commit' / 'No, won't fix' choice from you guys regarding this proposal (Patch included): AVRO-1633 Add additional setXxx(Builder) method to make user code more readable. Niels Basjes On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com wrote: Hi everyone, It would be good to do another Avro release soon. I'm happy to create a release candidate so we can vote on it. What do folks think about releasing 1.7.8 vs. 1.8.0? There are very few changes in the 1.7 branch, so it might be best to release 1.8.0 from trunk. There have been enough changes to justify a new minor release I think. Are there any drawbacks to doing that? The unresolved 1.8.0 issues are here: http://s.apache.org/Cdt And the unresolved 1.7.8 issues are here: http://s.apache.org/5ShO Thanks, Tom -- Best regards / Met vriendelijke groeten, Niels Basjes
Re: Next Avro release
On Wed, Jul 1, 2015 at 7:13 PM, Sean Busbey bus...@cloudera.com wrote: +1 for a 1.8.0 release. What kind of timeline for the first RC are you looking at? In the next week or so. Hopefully we can decide anything else we'd like to include and get it reviewed and committed in that time. On Wed, Jul 1, 2015 at 10:00 AM, Zoltan Farkas zolyfar...@yahoo.com.invalid wrote: I would like to have https://issues.apache.org/jira/browse/AVRO-1667 https://issues.apache.org/jira/browse/AVRO-1667 resolved in 1.8.0 if possible. This issue prevents serializing/deserializing objects with certain schemas… —Z On Jul 1, 2015, at 10:50 AM, Niels Basjes ni...@basjes.nl wrote: Hi, I would say go for the 1.8.0 Note that there are a few issues that seem appropriate to include for this one because they relate to a change in 'backwards compatibility: - AVRO-1586 Build against Hadoop 2 ( which should also fix AVRO-1453 Release version of avro-tools compiled against hadoop2 ) - AVRO-1559 Drop support for Ruby 1.8 On a personal note I would like a 'Yes, commit' / 'No, won't fix' choice from you guys regarding this proposal (Patch included): AVRO-1633 Add additional setXxx(Builder) method to make user code more readable. Niels Basjes On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com wrote: Hi everyone, It would be good to do another Avro release soon. I'm happy to create a release candidate so we can vote on it. What do folks think about releasing 1.7.8 vs. 1.8.0? There are very few changes in the 1.7 branch, so it might be best to release 1.8.0 from trunk. There have been enough changes to justify a new minor release I think. Are there any drawbacks to doing that? The unresolved 1.8.0 issues are here: http://s.apache.org/Cdt And the unresolved 1.7.8 issues are here: http://s.apache.org/5ShO Thanks, Tom -- Best regards / Met vriendelijke groeten, Niels Basjes -- Sean
[jira] [Commented] (AVRO-1667) Parser symbol tree flattening is broken for recursive schemas
[
https://issues.apache.org/jira/browse/AVRO-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610705#comment-14610705
]
Ryan Blue commented on AVRO-1667:
-
What I'm trying to understand is why the null productions are getting generated
(what is the underlying bug? what assumption did the previous version make that
was a problem?) and how your patch addresses it. I can look at the code to
figure it out, but it is usually faster if you can tell me what is going on.
Parser symbol tree flattening is broken for recursive schemas
-
Key: AVRO-1667
URL: https://issues.apache.org/jira/browse/AVRO-1667
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.7
Reporter: Zoltan Farkas
Here is a unit test to reproduce:
{noformat}
package org.apache.avro.io.parsing;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Set;
import junit.framework.Assert;
import org.apache.avro.Schema;
import org.junit.Test;
public class SymbolTest {
private static final String SCHEMA =
{\type\:\record\,\name\:\SampleNode\,
+ \namespace\:\org.spf4j.ssdump2.avro\,\n +
\fields\:[\n +
{\name\:\count\,\type\:\int\,\default\:0},\n +
{\name\:\subNodes\,\type\:\n +
{\type\:\array\,\items\:{\n +
\type\:\record\,\name\:\SamplePair\,\n +
\fields\:[\n +
{\name\:\method\,\type\:\n +
{\type\:\record\,\name\:\Method\,\n +
\fields\:[\n +
{\name\:\declaringClass\,\type\:{\type\:\string\,\avro.java.string\:\String\}},\n
+
{\name\:\methodName\,\type\:{\type\:\string\,\avro.java.string\:\String\}}\n
+
]}},\n +
{\name\:\node\,\type\:\SampleNode\}]}}}]};
@Test
public void testSomeMethod() throws IOException {
Schema schema = new Schema.Parser().parse(SCHEMA);
Symbol root = Symbol.root(new ResolvingGrammarGenerator()
.generate(schema, schema, new
HashMapValidatingGrammarGenerator.LitS, Symbol()));
validateNonNull(root, new HashSetSymbol());
}
private static void validateNonNull(final Symbol symb, SetSymbol seen) {
if (seen.contains(symb)) {
return;
} else {
seen.add(symb);
}
if (symb.production != null) {
for (Symbol s : symb.production) {
if (s == null) {
Assert.fail(invalid parsing tree should not contain
nulls);
}
if (s.kind != Symbol.Kind.ROOT) {
validateNonNull(s, seen);;
}
}
}
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
