Re: Radical idea for 1.9.0
(Thanks for the [Discuss] tip.) Recursion is primarily a code-maintenance problem. I'm not sure how to quantify the complexity, but certainly Schema.java itself has a lot of logic in it to deal with recursion, as do all the "Grammar Generator" classes plus Generic Data -- the classes that are performance sensitive. I don't think that recursion is inherently expensive performance-wise, but by making the encoding/resolution/decoding logic unnecessarily complicated, it makes it difficult to implement more aggressive strategies for higher-performance (e.g., dynamic code generation). On a similar note, the complexity of recursion could make it hard to add new features to Avro. I've yet to see compelling uses of recursion surface in this thread. Perhaps we deprecate recursion in 1.9, with the goal of eliminating it in 1.10? (Specifically, we write error messages to stderr when we parse recursive types -- with a flag to silence those message in case they get in someone's way.) If this deprecation creates howls of complaint because the feature is more useful than this thread seems to suggest, then we can keep it in. On Tue, Dec 4, 2018 at 9:25 AM Sean Busbey wrote: > > In the future please use "[DISCUSS]" at the start of your subject line > for these kinds of proposals. that'll get more folks to see the > discussion, e.g. when they filter this list. > > as a point of clarification, 1.9.0 is a major version change for the > Avro project. the "1" is a file format version. that's why API > incompatibilities are allowed in a new 1.y version. As Doug mentioned, > "would not be able to read binary files" means you're talking about a > file format incompatibility, which I don't think we should do. > > Are you interested in removing this feature to improve code > maintenance or to improve performance? both? > > Can you quantify the amount of complexity you're referring to? > On Fri, Nov 30, 2018 at 12:22 PM Raymie Stata wrote: > > > > I understand we've been willing to introduce backward-incompatible API > > changes (not file-format changes) into minor release versions. If so, > > here's an idea for consideration: > > > > Let's eliminate recursive records from Avro 1.9.x. Recursion > > introduces a _lot_ of complexity into many parts of the Avro code > > base. We could vastly simplify the code base, and probably speed > > things up, by getting rid of this feature. > > > > The specific proposal would be that Avro 1.9.x would refuse to accept > > recursive records, and thus would not be able to read binary files > > written by older versions of Avro. I haven't heard of anyone actually > > using them, so I don't think this would be a problem. > > > > -- > busbey
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715625#comment-16715625 ] ASF subversion and git services commented on AVRO-2069: --- Commit e65411f5cce2d58588fbb96a366560f6e6803dab in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=e65411f ] Squashed commit of the following: This closes #243 commit b2a5607578652f203e117c5b80d228618dd101f1 Merge: 500c28c2 54fc3f53 Author: Daniel Kulp Date: Mon Dec 3 14:46:47 2018 -0500 Merge branch 'master' into master commit 500c28c212d20e40fe48b6229cfc7764631f56e8 Author: Daniil Guitelson Date: Fri Sep 1 17:37:10 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code Fix broken unit tests in TestSpecificCompilerTool after migrating to org.junit.Test commit 35b891ebf868cea12f278641229807c1e2a35db2 Merge: c642adaf 189368ed Author: Daniil Gitelson Date: Fri Sep 1 17:33:13 2017 +0300 Merge pull request #1 from apache/master Sync up with origin repository commit c642adaf3d29183049cd4ad2e0012b002d95d7c6 Author: Daniil Guitelson Date: Wed Aug 30 18:37:41 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code Fix broken unit tests in TestSpecificCompilerTool commit 3414eae643517591015e3a6154c726c31f2d3095 Author: Daniil Guitelson Date: Tue Aug 29 12:13:42 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson >Priority: Major > Fix For: 1.9.0 > > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[
https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715627#comment-16715627
]
ASF GitHub Bot commented on AVRO-2069:
--
dkulp closed pull request #243: AVRO-2069: Use primitive fields in generated
getters & setters in Java code
URL: https://github.com/apache/avro/pull/243
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
b/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
index 25b4101a6..f2e3bcfe4 100644
---
a/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
+++
b/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
@@ -228,7 +228,7 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
#else * @return The value of the '${this.mangle($field.name(),
$schema.isError())}' field.
#end
*/
- public ${this.javaType($field.schema())} ${this.generateGetMethod($schema,
$field)}() {
+ public ${this.javaUnbox($field.schema())} ${this.generateGetMethod($schema,
$field)}() {
return ${this.mangle($field.name(), $schema.isError())};
}
#end
@@ -252,7 +252,7 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
#end
* @param value the value to set.
*/
- public void ${this.generateSetMethod($schema,
$field)}(${this.javaType($field.schema())} value) {
+ public void ${this.generateSetMethod($schema,
$field)}(${this.javaUnbox($field.schema())} value) {
this.${this.mangle($field.name(), $schema.isError())} = value;
}
#end
@@ -384,7 +384,7 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
#end
* @return The value.
*/
-public ${this.javaType($field.schema())} ${this.generateGetMethod($schema,
$field)}() {
+public ${this.javaUnbox($field.schema())}
${this.generateGetMethod($schema, $field)}() {
return ${this.mangle($field.name(), $schema.isError())};
}
diff --git
a/lang/java/ipc/src/test/java/org/apache/avro/message/TestCustomSchemaStore.java
b/lang/java/ipc/src/test/java/org/apache/avro/message/TestCustomSchemaStore.java
index ba960442a..5de9d6a38 100644
---
a/lang/java/ipc/src/test/java/org/apache/avro/message/TestCustomSchemaStore.java
+++
b/lang/java/ipc/src/test/java/org/apache/avro/message/TestCustomSchemaStore.java
@@ -62,7 +62,7 @@ public void testCompatibleReadWithSchemaFromSchemaStore()
throws Exception {
// Should work
assertEquals(nestedEvolve1.getRootName(), "RootName");
assertEquals(nestedEvolve1.getNested().getName(), "Name");
-assertEquals(nestedEvolve1.getNested().getValue(), Long.valueOf(1));
+assertEquals(nestedEvolve1.getNested().getValue(), 1);
}
@Test(expected = MissingSchemaException.class)
diff --git
a/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificRecordBuilder.java
b/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificRecordBuilder.java
index 4fa6fe0b7..1223e2325 100644
---
a/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificRecordBuilder.java
+++
b/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificRecordBuilder.java
@@ -47,7 +47,7 @@ public void testSpecificBuilder() {
Assert.assertTrue(builder.hasName());
Assert.assertEquals("James Gosling", builder.getName());
Assert.assertTrue(builder.hasYearOfBirth());
-Assert.assertEquals(new Integer(1955), builder.getYearOfBirth());
+Assert.assertEquals(1955, builder.getYearOfBirth());
Assert.assertFalse(builder.hasCountry());
Assert.assertNull(builder.getCountry());
Assert.assertTrue(builder.hasState());
@@ -102,13 +102,13 @@ public void testUnions() {
.setProduct(product)
.build())
.build();
-Assert.assertEquals(datetime, p.getDatetime().longValue());
+Assert.assertEquals(datetime, p.getDatetime());
Assert.assertEquals(ProductPage.class, p.getPageContext().getClass());
Assert.assertEquals(product,
((ProductPage)p.getPageContext()).getProduct());
PageView p2 = PageView.newBuilder(p).build();
-Assert.assertEquals(datetime, p2.getDatetime().longValue());
+Assert.assertEquals(datetime, p2.getDatetime());
Assert.assertEquals(ProductPage.class, p2.getPageContext().getClass());
Assert.assertEquals(product,
((ProductPage)p2.getPageContext()).getProduct());
@@ -141,10 +141,10 @@ public void testInterop() {
Assert.assertEquals(interop.getArrayField(),
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715624#comment-16715624 ] ASF subversion and git services commented on AVRO-2069: --- Commit e65411f5cce2d58588fbb96a366560f6e6803dab in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=e65411f ] Squashed commit of the following: This closes #243 commit b2a5607578652f203e117c5b80d228618dd101f1 Merge: 500c28c2 54fc3f53 Author: Daniel Kulp Date: Mon Dec 3 14:46:47 2018 -0500 Merge branch 'master' into master commit 500c28c212d20e40fe48b6229cfc7764631f56e8 Author: Daniil Guitelson Date: Fri Sep 1 17:37:10 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code Fix broken unit tests in TestSpecificCompilerTool after migrating to org.junit.Test commit 35b891ebf868cea12f278641229807c1e2a35db2 Merge: c642adaf 189368ed Author: Daniil Gitelson Date: Fri Sep 1 17:33:13 2017 +0300 Merge pull request #1 from apache/master Sync up with origin repository commit c642adaf3d29183049cd4ad2e0012b002d95d7c6 Author: Daniil Guitelson Date: Wed Aug 30 18:37:41 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code Fix broken unit tests in TestSpecificCompilerTool commit 3414eae643517591015e3a6154c726c31f2d3095 Author: Daniil Guitelson Date: Tue Aug 29 12:13:42 2017 +0300 AVRO-2069: Use primitive fields in generated getters & setters in Java code > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson >Priority: Major > Fix For: 1.9.0 > > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2069: -- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson >Priority: Major > Fix For: 1.9.0 > > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2004) Stats View is not working
[ https://issues.apache.org/jira/browse/AVRO-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2004: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Stats View is not working > - > > Key: AVRO-2004 > URL: https://issues.apache.org/jira/browse/AVRO-2004 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Ken Dombeck >Assignee: Ken Dombeck >Priority: Minor > Fix For: 1.9.0 > > Attachments: WorkingStatsPage.png > > > When trying to view the IPC stats page the charts are not being displayed. > The issue is that the CSS and JS resources are not being included in the jar > artifact. > There are 2 possible solutions that I see to this. > # Move the CSS and JS resources to be with the Velocity template that > generates the HTML. > # Change Maven to include the resources where they currently are. > Github [pull request|https://github.com/apache/avro/pull/202] has been > created for option 1. > This relates to the feature created in > [AVRO-587|https://issues.apache.org/jira/browse/AVRO-587] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715598#comment-16715598
]
ASF GitHub Bot commented on AVRO-2058:
--
samschlegel commented on issue #237: [AVRO-2058] ReflectData#isNonStringMap
returns true for Utf8 keys
URL: https://github.com/apache/avro/pull/237#issuecomment-445983786
Ah, thanks @dkulp! Sorry this got away from me :sweat_smile:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Fix For: 1.9.0
>
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2004) Stats View is not working
[ https://issues.apache.org/jira/browse/AVRO-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715550#comment-16715550 ] ASF subversion and git services commented on AVRO-2004: --- Commit 67dbda933bb2e1286a28fd832ea4316d34906005 in avro's branch refs/heads/master from [~kdombeck] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=67dbda9 ] AVRO-2004: moved CSS and JS files to a directory where they will be included in the final jar > Stats View is not working > - > > Key: AVRO-2004 > URL: https://issues.apache.org/jira/browse/AVRO-2004 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Ken Dombeck >Assignee: Ken Dombeck >Priority: Minor > Fix For: 1.9.0 > > Attachments: WorkingStatsPage.png > > > When trying to view the IPC stats page the charts are not being displayed. > The issue is that the CSS and JS resources are not being included in the jar > artifact. > There are 2 possible solutions that I see to this. > # Move the CSS and JS resources to be with the Velocity template that > generates the HTML. > # Change Maven to include the resources where they currently are. > Github [pull request|https://github.com/apache/avro/pull/202] has been > created for option 1. > This relates to the feature created in > [AVRO-587|https://issues.apache.org/jira/browse/AVRO-587] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2004) Stats View is not working
[
https://issues.apache.org/jira/browse/AVRO-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715549#comment-16715549
]
ASF GitHub Bot commented on AVRO-2004:
--
dkulp closed pull request #202: AVRO-2004: fix IPC charts for stats
URL: https://github.com/apache/avro/pull/202
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/avro.css
b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/avro.css
similarity index 100%
rename from
lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/avro.css
rename to
lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/avro.css
diff --git
a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/avro.js
b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/avro.js
similarity index 81%
rename from lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/avro.js
rename to
lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/avro.js
index 30da1b017..1d0f09772 100644
--- a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/avro.js
+++ b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/avro.js
@@ -44,7 +44,7 @@ function makeDotChart(yVals) {
vis.add(pv.Rule)
.data(y.ticks())
.bottom(y)
- .strokeStyle(function(d) d ? "#eee" : "#000")
+ .strokeStyle(function(d) {return d ? "#eee" : "#000"})
.anchor("left").add(pv.Label)
.text(y.tickFormat);
@@ -53,11 +53,11 @@ function makeDotChart(yVals) {
vis.add(pv.Panel)
.data(data)
.add(pv.Dot)
- .left(function(d) x(d.x))
- .bottom(function(d) y(d.y))
- .strokeStyle(function(d) dotColors[d.x % 20])
- .fillStyle(function() this.strokeStyle().alpha(1))
- .title(function(d) d.y)
+ .left(function(d) {return x(d.x)})
+ .bottom(function(d) {return y(d.y)})
+ .strokeStyle(function(d) {return dotColors[d.x % 20]})
+ .fillStyle(function() {return this.strokeStyle().alpha(1)})
+ .title(function(d) {return d.y})
.event("mouseover", pv.Behavior.tipsy({gravity: "n",
fade: false, delayIn: 0}));
vis.render();
@@ -86,7 +86,7 @@ function makeBarChart(labels, boundries, data) {
.width(10)
.bottom(0)
.height(y)
- .fillStyle(function(d) "#1f77b4")
+ .fillStyle(function(d) {return "#1f77b4"})
.title(function() { return boundries[this.index]; })
.event("mouseover", pv.Behavior.tipsy({gravity: "n",
fade: false, delayIn: 0}));
@@ -94,17 +94,17 @@ function makeBarChart(labels, boundries, data) {
bar.anchor("bottom").add(pv.Label)
.textMargin(5)
.textBaseline("top")
- .text(function() (this.index % 4 == 0) ? labels[this.index]:
"");
+ .text(function() {return (this.index % 4 == 0) ?
labels[this.index]: ""});
vis.add(pv.Rule)
.data(y.ticks())
- .bottom(function(d) Math.round(y(d)) - .5)
- .strokeStyle(function(d) d ? "rgba(255,255,255,.3)" : "#000")
+ .bottom(function(d) {return Math.round(y(d)) - .5})
+ .strokeStyle(function(d) {return d ? "rgba(255,255,255,.3)" :
"#000"})
.add(pv.Rule)
.left(0)
.width(5)
.strokeStyle("#000")
.anchor("left").add(pv.Label)
- .text(function(d) d.toFixed(1));
+ .text(function(d) {return d.toFixed(1)});
vis.render();
-}
\ No newline at end of file
+}
diff --git
a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/g.bar.js
b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/g.bar.js
similarity index 100%
rename from
lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/g.bar.js
rename to
lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/g.bar.js
diff --git
a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/jquery-1.4.2.min.js
b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/jquery-1.4.2.min.js
similarity index 100%
rename from
lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/jquery-1.4.2.min.js
rename to
lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/jquery-1.4.2.min.js
diff --git
a/lang/java/ipc/src/main/java/org/apache/avro/ipc/stats/static/jquery.tipsy.js
b/lang/java/ipc/src/main/velocity/org/apache/avro/ipc/stats/static/jquery.tipsy.js
similarity index 100%
rename from
[jira] [Updated] (AVRO-2007) for avro-c enable getting enum size (i.e. number of symbols at enum schema)
[ https://issues.apache.org/jira/browse/AVRO-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2007: -- Resolution: Duplicate Fix Version/s: (was: 1.8.1) 1.9.0 Status: Resolved (was: Patch Available) Duplicate of AVRO-1528 > for avro-c enable getting enum size (i.e. number of symbols at enum schema) > --- > > Key: AVRO-2007 > URL: https://issues.apache.org/jira/browse/AVRO-2007 > Project: Apache Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.8.1 >Reporter: Eliyahu Machluf >Priority: Minor > Fix For: 1.9.0 > > > Avro-c library stores internaly enum as int value, there is no validation, > when setting an enum value, it match the schema. > Therefore, I want to enable getting enum size (i.e. number of symbols at enum > schema), in order to enable validating (when using the instance) that the > value is at enum range. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2007) for avro-c enable getting enum size (i.e. number of symbols at enum schema)
[
https://issues.apache.org/jira/browse/AVRO-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715543#comment-16715543
]
ASF GitHub Bot commented on AVRO-2007:
--
dkulp closed pull request #167: AVRO-2007: enable getting schema enum size
(i.e. number of symbols at enum schema).
URL: https://github.com/apache/avro/pull/167
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/src/avro/schema.h b/lang/c/src/avro/schema.h
index 41dd8ea48..0c67deda8 100644
--- a/lang/c/src/avro/schema.h
+++ b/lang/c/src/avro/schema.h
@@ -61,6 +61,7 @@ int avro_schema_enum_get_by_name(const avro_schema_t enump,
const char *symbol_name);
int avro_schema_enum_symbol_append(const avro_schema_t
enump, const char *symbol);
+size_t avro_schema_enum_size(const avro_schema_t enump);
avro_schema_t avro_schema_fixed(const char *name, const int64_t len);
avro_schema_t avro_schema_fixed_ns(const char *name, const char *space,
diff --git a/lang/c/src/schema.c b/lang/c/src/schema.c
index 3ade1140e..f1968e1e3 100644
--- a/lang/c/src/schema.c
+++ b/lang/c/src/schema.c
@@ -574,6 +574,11 @@ avro_schema_enum_symbol_append(const avro_schema_t
enum_schema,
return 0;
}
+size_t avro_schema_enum_size( const avro_schema_t enump )
+{
+ return avro_schema_to_enum(enump)->symbols->num_entries;
+}
+
int
avro_schema_record_field_append(const avro_schema_t record_schema,
const char *field_name,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> for avro-c enable getting enum size (i.e. number of symbols at enum schema)
> ---
>
> Key: AVRO-2007
> URL: https://issues.apache.org/jira/browse/AVRO-2007
> Project: Apache Avro
> Issue Type: Improvement
> Components: c
>Affects Versions: 1.8.1
>Reporter: Eliyahu Machluf
>Priority: Minor
> Fix For: 1.9.0
>
>
> Avro-c library stores internaly enum as int value, there is no validation,
> when setting an enum value, it match the schema.
> Therefore, I want to enable getting enum size (i.e. number of symbols at enum
> schema), in order to enable validating (when using the instance) that the
> value is at enum range.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2007) for avro-c enable getting enum size (i.e. number of symbols at enum schema)
[ https://issues.apache.org/jira/browse/AVRO-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715542#comment-16715542 ] ASF GitHub Bot commented on AVRO-2007: -- dkulp commented on issue #167: AVRO-2007: enable getting schema enum size (i.e. number of symbols at enum schema). URL: https://github.com/apache/avro/pull/167#issuecomment-445968320 Duplicate of AVRO-1528 which was already applied. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > for avro-c enable getting enum size (i.e. number of symbols at enum schema) > --- > > Key: AVRO-2007 > URL: https://issues.apache.org/jira/browse/AVRO-2007 > Project: Apache Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.8.1 >Reporter: Eliyahu Machluf >Priority: Minor > Fix For: 1.9.0 > > > Avro-c library stores internaly enum as int value, there is no validation, > when setting an enum value, it match the schema. > Therefore, I want to enable getting enum size (i.e. number of symbols at enum > schema), in order to enable validating (when using the instance) that the > value is at enum range. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1844) Avro-C build procedure doesn't set include paths for Jansson
[ https://issues.apache.org/jira/browse/AVRO-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1844. --- Resolution: Fixed Fix Version/s: 1.9.0 > Avro-C build procedure doesn't set include paths for Jansson > > > Key: AVRO-1844 > URL: https://issues.apache.org/jira/browse/AVRO-1844 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.0 > Environment: gcc 4.9.3, Scientific Linux Fermi release 6.7 >Reporter: Jim Pivarski >Priority: Major > Fix For: 1.9.0 > > > I `configure` Jansson with a user-space `PREFIX` because I don't have root > access, and when I `make install`, the Jansson includes and libs appear in > new `include` and `lib` directories, as expected. > I `cmake` with `-DCMAKE_INSTALL_PREFIX:PATH=` the same path as `PREFIX` and > CMake is successful: it finds Jansson and makes the Makefiles. > BUT, when I `make`, it fails because it can't find `jansson.h`. The same is > true if I explicitly set `PKG_CONFIG_PATH` to point to > `$PREFIX/lib/pkgconfig` and `pkg-config jansson --cflags --libs` works. > If I explicitly add full paths to Avro's and Jansson's `#include ` > lines in the source code, Avro compiles without trouble. > (`avro-c-1.8.0/src/schema.c`, `avro-c-1.8.0/src/value-json.c`, > `include/jansson.h`) > Presumably, the Jansson part of the build process is only partly implemented, > such that the CMake script searches for the appropriate version but `-I` > flags are not generated for the compiler. I don't know enough CMake to know > where to insert these. > Fixing this issue is the only way I know of to make Avro-C installable on a > system without root access, apart from the hack I described above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1844) Avro-C build procedure doesn't set include paths for Jansson
[
https://issues.apache.org/jira/browse/AVRO-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715531#comment-16715531
]
ASF GitHub Bot commented on AVRO-1844:
--
dkulp closed pull request #225: AVRO-1844: set correct header searching path
for zlib, snappy & jansson.
URL: https://github.com/apache/avro/pull/225
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/CMakeLists.txt b/lang/c/CMakeLists.txt
index fda4382a2..b8c94847b 100644
--- a/lang/c/CMakeLists.txt
+++ b/lang/c/CMakeLists.txt
@@ -131,7 +131,7 @@ find_package(ZLIB)
if (ZLIB_FOUND)
set(ZLIB_PKG zlib)
add_definitions(-DDEFLATE_CODEC)
-include_directories(${ZLIB_INCLUDE_DIR})
+include_directories(${ZLIB_INCLUDE_DIRS})
message("Enabled deflate codec")
else (ZLIB_FOUND)
set(ZLIB_PKG "")
@@ -143,7 +143,7 @@ find_package(Snappy)
if (SNAPPY_FOUND AND ZLIB_FOUND) # Snappy borrows crc32 from zlib
set(SNAPPY_PKG libsnappy)
add_definitions(-DSNAPPY_CODEC)
-include_directories(${SNAPPY_INCLUDE_DIR})
+include_directories(${SNAPPY_INCLUDE_DIRS})
message("Enabled snappy codec")
else (SNAPPY_FOUND AND ZLIB_FOUND)
set(SNAPPY_PKG "")
@@ -172,7 +172,7 @@ set(CODEC_PKG "@ZLIB_PKG@ @LZMA_PKG@ @SNAPPY_PKG@")
pkg_check_modules(JANSSON jansson>=2.3)
if (JANSSON_FOUND)
set(JANSSON_PKG libjansson)
-include_directories(${JANSSON_INCLUDE_DIR})
+include_directories(${JANSSON_INCLUDE_DIRS})
link_directories(${JANSSON_LIBRARY_DIRS})
else (JANSSON_FOUND)
message(FATAL_ERROR "libjansson >=2.3 not found")
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Avro-C build procedure doesn't set include paths for Jansson
>
>
> Key: AVRO-1844
> URL: https://issues.apache.org/jira/browse/AVRO-1844
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Affects Versions: 1.8.0
> Environment: gcc 4.9.3, Scientific Linux Fermi release 6.7
>Reporter: Jim Pivarski
>Priority: Major
> Fix For: 1.9.0
>
>
> I `configure` Jansson with a user-space `PREFIX` because I don't have root
> access, and when I `make install`, the Jansson includes and libs appear in
> new `include` and `lib` directories, as expected.
> I `cmake` with `-DCMAKE_INSTALL_PREFIX:PATH=` the same path as `PREFIX` and
> CMake is successful: it finds Jansson and makes the Makefiles.
> BUT, when I `make`, it fails because it can't find `jansson.h`. The same is
> true if I explicitly set `PKG_CONFIG_PATH` to point to
> `$PREFIX/lib/pkgconfig` and `pkg-config jansson --cflags --libs` works.
> If I explicitly add full paths to Avro's and Jansson's `#include `
> lines in the source code, Avro compiles without trouble.
> (`avro-c-1.8.0/src/schema.c`, `avro-c-1.8.0/src/value-json.c`,
> `include/jansson.h`)
> Presumably, the Jansson part of the build process is only partly implemented,
> such that the CMake script searches for the appropriate version but `-I`
> flags are not generated for the compiler. I don't know enough CMake to know
> where to insert these.
> Fixing this issue is the only way I know of to make Avro-C installable on a
> system without root access, apart from the hack I described above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2058:
--
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Fix For: 1.9.0
>
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715500#comment-16715500
]
ASF subversion and git services commented on AVRO-2058:
---
Commit 269d0ff7d3025cf3370023e353a373916dd67433 in avro's branch
refs/heads/master from [~dkulp]
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=269d0ff ]
[AVRO-2058] Fix issues with Utf8 keys maps being treated as non-strings
This closes #237
commit 8852da40e7a0ee58d9b027173a4972e7e71b432d
Author: Sam Schlegel
Date: Wed Jul 26 19:20:51 2017 -0700
Add failing test
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2008) change the error module to be thread-safe on Windows as well as on UNIX
[ https://issues.apache.org/jira/browse/AVRO-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2008: -- Resolution: Fixed Fix Version/s: (was: 1.8.3) 1.9.0 Status: Resolved (was: Patch Available) > change the error module to be thread-safe on Windows as well as on UNIX > --- > > Key: AVRO-2008 > URL: https://issues.apache.org/jira/browse/AVRO-2008 > Project: Apache Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.8.1 >Reporter: Eliyahu Machluf >Assignee: Eliyahu Machluf >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2008) change the error module to be thread-safe on Windows as well as on UNIX
[
https://issues.apache.org/jira/browse/AVRO-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715472#comment-16715472
]
ASF GitHub Bot commented on AVRO-2008:
--
dkulp closed pull request #164: AVRO-2008: change the error module to be
thread-safe on Windows as well as on UNIX
URL: https://github.com/apache/avro/pull/164
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/src/errors.c b/lang/c/src/errors.c
index a033be74d..23934a6e7 100644
--- a/lang/c/src/errors.c
+++ b/lang/c/src/errors.c
@@ -22,17 +22,6 @@
#include "avro/errors.h"
-#if defined THREADSAFE && (defined __unix__ || defined __unix)
-#include
-static pthread_key_t error_data_key;
-static pthread_once_t error_data_key_once = PTHREAD_ONCE_INIT;
-
-static void make_error_data_key()
-{
-pthread_key_create(_data_key, free);
-}
-#endif
-
/* 4K should be enough, right? */
#define AVRO_ERROR_SIZE 4096
@@ -52,10 +41,30 @@ struct avro_error_data_t {
};
+#if defined THREADSAFE
+#if ( defined __unix__ || defined __unix )
+#include
+static pthread_key_t error_data_key;
+static pthread_once_t error_data_key_once = PTHREAD_ONCE_INIT;
+
+static void make_error_data_key()
+{
+pthread_key_create(_data_key, free);
+}
+#elif defined _WIN32
+#include
+
+static __declspec( thread ) struct avro_error_data_t TLS_ERROR_DATA = { "",
"", NULL, NULL };
+
+#endif /* unix||_unix||_WIN32 */
+#endif /* THREADSAFE */
+
static struct avro_error_data_t *
avro_get_error_data(void)
{
-#if defined THREADSAFE && (defined __unix__ || defined __unix)
+#if defined THREADSAFE
+#if defined __unix__ || defined __unix
+
pthread_once(_data_key_once, make_error_data_key);
struct avro_error_data_t *ERROR_DATA =
@@ -72,7 +81,20 @@ avro_get_error_data(void)
}
return ERROR_DATA;
-#else
+
+#elif defined _WIN32
+
+ if ( TLS_ERROR_DATA.AVRO_CURRENT_ERROR == NULL )
+ {
+ //first usage of the ERROR_DATA, initialize 'current' and
'other' pointers.
+ TLS_ERROR_DATA.AVRO_CURRENT_ERROR = TLS_ERROR_DATA.AVRO_ERROR1;
+ TLS_ERROR_DATA.AVRO_OTHER_ERROR = TLS_ERROR_DATA.AVRO_ERROR2;
+ }
+ return _ERROR_DATA;
+
+ #endif /* UNIX and WIN32 threadsafe handling */
+
+#else /* not thread-safe */
static struct avro_error_data_t ERROR_DATA = {
/* .AVRO_ERROR1 = */ {'\0'},
/* .AVRO_ERROR2 = */ {'\0'},
diff --git a/lang/c/tests/test_avro_errors_are_thread_safe.c
b/lang/c/tests/test_avro_errors_are_thread_safe.c
new file mode 100644
index 0..8e2f70d0f
--- /dev/null
+++ b/lang/c/tests/test_avro_errors_are_thread_safe.c
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License.
+ */
+
+#ifdef _WIN32
+#include
+#define THR_HANDLE HANDLE
+#define LAST_ERROR() GetLastError()
+#define SLEEP_MILLIS(millis) Sleep( millis )
+#else
+#include
+#include
+#include
+#define THR_HANDLE pthread_t
+#define LAST_ERROR() errno
+#define SLEEP_MILLIS(millis) usleep( millis * 1000 )
+#endif
+#include
+#include
+#include
+#include
+#include
+
+enum {
+ NUMBER_OF_TEST_THREADS = 64,
+ THREAD_ERROR_BUFFER_SIZE = 1024,
+ MAX_THREAD_SLEEP_MILLIS = 3000,
+};
+
+typedef struct _TEST_THREAD_DATA
+{
+ int index;
+ int error_occured;
+ char error_message[THREAD_ERROR_BUFFER_SIZE];
+ volatile int worker_thread;
+ int sleep_interval_millis;
+}TEST_THREAD_DATA;
+
+static int get_random_value( int max_value );
+#ifdef _WIN32
+static DWORD worker_thread( LPVOID lpThreadParameter );
+#else
+static void *worker_thread( void *p );
+#endif
+static THR_HANDLE launch_thread( void *thread_context );
+static int join_thread( THR_HANDLE thread_handle );
+
+int main()
+{
+ TEST_THREAD_DATA threads_data[NUMBER_OF_TEST_THREADS];
+ THR_HANDLE thread_handle[NUMBER_OF_TEST_THREADS];
+ unsigned i;
+ int found_error = 0;
+
+ srand( (unsigned)time( NULL
[jira] [Commented] (AVRO-2063) C: VERSION, SOVERSION not set correctly for shared library
[
https://issues.apache.org/jira/browse/AVRO-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715415#comment-16715415
]
ASF GitHub Bot commented on AVRO-2063:
--
dkulp closed pull request #239: AVRO-2063. C: VERSION, SOVERSION not set
correctly for shared library
URL: https://github.com/apache/avro/pull/239
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/CMakeLists.txt b/lang/c/CMakeLists.txt
index fda4382a2..4161de244 100644
--- a/lang/c/CMakeLists.txt
+++ b/lang/c/CMakeLists.txt
@@ -47,6 +47,16 @@ if (UNIX)
if(LIBAVRO_VERSION_RESULT)
message(FATAL_ERROR "Cannot determine libavro version number")
endif(LIBAVRO_VERSION_RESULT)
+
+execute_process(COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/version.sh libcurrent
+WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+RESULT_VARIABLE LIBAVRO_SOVERSION_RESULT
+OUTPUT_VARIABLE LIBAVRO_SOVERSION
+OUTPUT_STRIP_TRAILING_WHITESPACE)
+if(LIBAVRO_SOVERSION_RESULT)
+message(FATAL_ERROR "Cannot determine libavro version number")
+endif(LIBAVRO_SOVERSION_RESULT)
+
else(UNIX)
# Hard code for win32 -- need to figure out how to port version.sh for
# Windows.
diff --git a/lang/c/src/CMakeLists.txt b/lang/c/src/CMakeLists.txt
index c21f1ce08..330d5466d 100644
--- a/lang/c/src/CMakeLists.txt
+++ b/lang/c/src/CMakeLists.txt
@@ -79,7 +79,7 @@ set(AVRO_SRC
source_group(Avro FILES ${AVRO_SRC})
-# The version.sh script gives us a SOVERSION that uses colon as a
+# The version.sh script gives us a VERSION that uses colon as a
# separator; we need periods.
string(REPLACE ":" "." LIBAVRO_DOT_VERSION ${LIBAVRO_VERSION})
@@ -94,7 +94,8 @@ add_library(avro-shared SHARED ${AVRO_SRC})
target_link_libraries(avro-shared ${JANSSON_LIBRARIES} ${CODEC_LIBRARIES}
${THREADS_LIBRARIES})
set_target_properties(avro-shared PROPERTIES
OUTPUT_NAME avro
-SOVERSION ${LIBAVRO_DOT_VERSION})
+VERSION ${LIBAVRO_DOT_VERSION}
+SOVERSION ${LIBAVRO_SOVERSION})
endif(NOT WIN32)
install(FILES
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C: VERSION, SOVERSION not set correctly for shared library
> --
>
> Key: AVRO-2063
> URL: https://issues.apache.org/jira/browse/AVRO-2063
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Affects Versions: 1.8.2
>Reporter: Robert Edmonds
>Priority: Minor
> Fix For: 1.9.0
>
>
> Hi,
> I'm attempting to package avro-c for Debian. While doing so, I noticed that
> the libavro.so symlink that gets installed points directly to the
> libavro.so.23.0.0 library file, without an intermediate "libavro.so.23"
> symlink, and that the SONAME embedded in the library file is set to
> "libavro.so.23.0.0", rather than "libavro.so.23", which is what I would
> expect for a library trying to follow the libtool-style current/revision/age
> scheme on Linux systems.
> It looks like the recommended way to get CMake to produce the typical symlink
> chain and SONAME is to set the VERSION property to "23.0.0" and the SOVERSION
> to 23 on the shared library target. See this article for details:
> http://pusling.com/blog/?p=352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2026) Incorporate valgrind memory testing for C
[ https://issues.apache.org/jira/browse/AVRO-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2026: -- Resolution: Fixed Fix Version/s: (was: 1.8.3) 1.9.0 Status: Resolved (was: Patch Available) > Incorporate valgrind memory testing for C > - > > Key: AVRO-2026 > URL: https://issues.apache.org/jira/browse/AVRO-2026 > Project: Apache Avro > Issue Type: Improvement > Components: c >Reporter: John Gill >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-2026.patch > > > With the work done for AVRO-1167 & AVRO-766 in identifying memory leaks > decided to make it a standard part of the testing framework. This issue > requires https://github.com/apache/avro/pull/218 otherwise test_avro_schema > and test_avro_values fail due to to avro_schema_copy calls. > Adding valgrind tool to the docker container > Adding valgrind to CMakeLists.txt to optionally do valgrind testing if the > tool is found > Cleaning up existing tests that were failing due to not properly cleaning up > memory > Lastly, as part of ./build.sh test for lang/c do not remove the build/c > directory. The memcheck output is kept so that memory leak issues can be > reviewed and addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2063) C: VERSION, SOVERSION not set correctly for shared library
[ https://issues.apache.org/jira/browse/AVRO-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715417#comment-16715417 ] ASF subversion and git services commented on AVRO-2063: --- Commit 4b1ab225fef8e8f66cb95d6605bdd4fd568929ed in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=4b1ab22 ] AVRO-2063. C: VERSION, SOVERSION not set correctly for shared library (#239) Prior to this commit the VERSION and SOVERSION properties were not being set correctly for the Avro C shared library. Previously, VERSION was unset, and SOVERSION was set to the full dotted version string (e.g. "23.0.0"), which resulted in a SONAME of "libavro.so.23.0.0", which is incorrect. This commit sets SOVERSION to the library's "current" version number (obtained by invoking "version.sh libcurrent"), and sets VERSION to the full dotted version string. This should result in a proper libavro.so → libavro.so.23 → libavro.so.23.0.0 symlink chain with a SONAME of "libavro.so.23". See Sune Vuorela's article on "CMake and library properties": http://pusling.com/blog/?p=352. > C: VERSION, SOVERSION not set correctly for shared library > -- > > Key: AVRO-2063 > URL: https://issues.apache.org/jira/browse/AVRO-2063 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.2 >Reporter: Robert Edmonds >Priority: Minor > Fix For: 1.9.0 > > > Hi, > I'm attempting to package avro-c for Debian. While doing so, I noticed that > the libavro.so symlink that gets installed points directly to the > libavro.so.23.0.0 library file, without an intermediate "libavro.so.23" > symlink, and that the SONAME embedded in the library file is set to > "libavro.so.23.0.0", rather than "libavro.so.23", which is what I would > expect for a library trying to follow the libtool-style current/revision/age > scheme on Linux systems. > It looks like the recommended way to get CMake to produce the typical symlink > chain and SONAME is to set the VERSION property to "23.0.0" and the SOVERSION > to 23 on the shared library target. See this article for details: > http://pusling.com/blog/?p=352. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2063) C: VERSION, SOVERSION not set correctly for shared library
[ https://issues.apache.org/jira/browse/AVRO-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715416#comment-16715416 ] ASF subversion and git services commented on AVRO-2063: --- Commit 5e852265ea7197efe809a8c37ec6a0c34811d3ba in avro's branch refs/heads/master from [~edmonds] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=5e85226 ] AVRO-2063. C: VERSION, SOVERSION not set correctly for shared library Prior to this commit the VERSION and SOVERSION properties were not being set correctly for the Avro C shared library. Previously, VERSION was unset, and SOVERSION was set to the full dotted version string (e.g. "23.0.0"), which resulted in a SONAME of "libavro.so.23.0.0", which is incorrect. This commit sets SOVERSION to the library's "current" version number (obtained by invoking "version.sh libcurrent"), and sets VERSION to the full dotted version string. This should result in a proper libavro.so → libavro.so.23 → libavro.so.23.0.0 symlink chain with a SONAME of "libavro.so.23". See Sune Vuorela's article on "CMake and library properties": http://pusling.com/blog/?p=352. > C: VERSION, SOVERSION not set correctly for shared library > -- > > Key: AVRO-2063 > URL: https://issues.apache.org/jira/browse/AVRO-2063 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.2 >Reporter: Robert Edmonds >Priority: Minor > Fix For: 1.9.0 > > > Hi, > I'm attempting to package avro-c for Debian. While doing so, I noticed that > the libavro.so symlink that gets installed points directly to the > libavro.so.23.0.0 library file, without an intermediate "libavro.so.23" > symlink, and that the SONAME embedded in the library file is set to > "libavro.so.23.0.0", rather than "libavro.so.23", which is what I would > expect for a library trying to follow the libtool-style current/revision/age > scheme on Linux systems. > It looks like the recommended way to get CMake to produce the typical symlink > chain and SONAME is to set the VERSION property to "23.0.0" and the SOVERSION > to 23 on the shared library target. See this article for details: > http://pusling.com/blog/?p=352. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2063) C: VERSION, SOVERSION not set correctly for shared library
[ https://issues.apache.org/jira/browse/AVRO-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2063. --- Resolution: Fixed Fix Version/s: 1.9.0 > C: VERSION, SOVERSION not set correctly for shared library > -- > > Key: AVRO-2063 > URL: https://issues.apache.org/jira/browse/AVRO-2063 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.2 >Reporter: Robert Edmonds >Priority: Minor > Fix For: 1.9.0 > > > Hi, > I'm attempting to package avro-c for Debian. While doing so, I noticed that > the libavro.so symlink that gets installed points directly to the > libavro.so.23.0.0 library file, without an intermediate "libavro.so.23" > symlink, and that the SONAME embedded in the library file is set to > "libavro.so.23.0.0", rather than "libavro.so.23", which is what I would > expect for a library trying to follow the libtool-style current/revision/age > scheme on Linux systems. > It looks like the recommended way to get CMake to produce the typical symlink > chain and SONAME is to set the VERSION property to "23.0.0" and the SOVERSION > to 23 on the shared library target. See this article for details: > http://pusling.com/blog/?p=352. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2026) Incorporate valgrind memory testing for C
[
https://issues.apache.org/jira/browse/AVRO-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715406#comment-16715406
]
ASF GitHub Bot commented on AVRO-2026:
--
dkulp closed pull request #218: AVRO-2026: Add valgrind to c tests
URL: https://github.com/apache/avro/pull/218
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/build.sh b/lang/c/build.sh
index f76f03550..cf05c1367 100755
--- a/lang/c/build.sh
+++ b/lang/c/build.sh
@@ -61,7 +61,6 @@ case "$1" in
prepare_build
make -C $build_dir
make -C $build_dir test
-clean
;;
dist)
diff --git a/lang/c/tests/CMakeLists.txt b/lang/c/tests/CMakeLists.txt
index 445e689a7..b885d28ca 100644
--- a/lang/c/tests/CMakeLists.txt
+++ b/lang/c/tests/CMakeLists.txt
@@ -40,27 +40,45 @@ macro(add_avro_test name)
)
endmacro(add_avro_test)
+macro(add_avro_test_checkmem name)
+add_avro_test(${name} ${ARGN})
+if(UNIX)
+find_program(MEMORYCHECK_COMMAND valgrind )
+if(MEMORYCHECK_COMMAND)
+add_test(memcheck_${name}
+${CMAKE_COMMAND} -E chdir ${AvroC_SOURCE_DIR}/tests
+${MEMORYCHECK_COMMAND}
+--log-file=${CMAKE_CURRENT_BINARY_DIR}/memcheck_${name}.log
+--leak-check=full
+--show-reachable=yes
+--error-exitcode=1
+${exec_name}
+)
+endif(MEMORYCHECK_COMMAND)
+endif (UNIX)
+endmacro(add_avro_test_checkmem)
+
add_avro_executable(generate_interop_data)
add_avro_executable(performance)
add_avro_executable(test_interop_data)
-add_avro_test(test_data_structures)
-add_avro_test(test_avro_schema)
-add_avro_test(test_avro_schema_names)
-add_avro_test(test_avro_values)
-add_avro_test(test_avro_968)
-add_avro_test(test_avro_984)
-add_avro_test(test_avro_1034)
-add_avro_test(test_avro_1084)
-add_avro_test(test_avro_1087)
-add_avro_test(test_avro_1165)
-add_avro_test(test_avro_1237)
-add_avro_test(test_avro_1238)
-add_avro_test(test_avro_1279)
-add_avro_test(test_avro_1405)
-add_avro_test(test_avro_1572)
-add_avro_test(test_avro_data)
-add_avro_test(test_refcount)
-add_avro_test(test_cpp test_cpp.cpp)
-add_avro_test(test_avro_1379)
-add_avro_test(test_avro_1691)
+add_avro_test_checkmem(test_data_structures)
+add_avro_test_checkmem(test_avro_schema)
+add_avro_test_checkmem(test_avro_schema_names)
+add_avro_test_checkmem(test_avro_values)
+add_avro_test_checkmem(test_avro_968)
+add_avro_test_checkmem(test_avro_984)
+add_avro_test_checkmem(test_avro_1034)
+add_avro_test_checkmem(test_avro_1084)
+add_avro_test_checkmem(test_avro_1087)
+add_avro_test_checkmem(test_avro_1165)
+add_avro_test_checkmem(test_avro_1237)
+add_avro_test_checkmem(test_avro_1238)
+add_avro_test_checkmem(test_avro_1279)
+add_avro_test_checkmem(test_avro_1405)
+add_avro_test_checkmem(test_avro_1572)
+add_avro_test(test_avro_data) # Skip memory check for datum. Deprecated and
has a lot of memory issues
+add_avro_test_checkmem(test_refcount)
+add_avro_test_checkmem(test_cpp test_cpp.cpp)
+add_avro_test_checkmem(test_avro_1379)
+add_avro_test_checkmem(test_avro_1691)
diff --git a/lang/c/tests/test_avro_1034.c b/lang/c/tests/test_avro_1034.c
index 036337ae4..957f30158 100644
--- a/lang/c/tests/test_avro_1034.c
+++ b/lang/c/tests/test_avro_1034.c
@@ -37,18 +37,18 @@
* resolved_reader_class and a corresponding resolved_record instance
* is created (using identical "writer" and "reader" schemas for
* simplicity), and an attempt is made to "read" the resolved avro
- * value.
- *
+ * value.
+ *
* Once the resolved value has been read, the source value (nested)
* and the resolved value (resolved_record) are both reset using
* avro_value_reset(). Then the source value (nested) is populated
* with another (larger) nested array. Then an attempt is made to read
* the resolved avro value again.
- *
+ *
* This second attempt to read the resolved value results in a
- * segmentation fault under Linux, using the patch in
+ * segmentation fault under Linux, using the patch in
*
https://issues.apache.org/jira/secure/attachment/12516487/0001-AVRO-1034.-C-Resolved-reader-initializes-child-array.patch.
- *
+ *
* However, the program does not seg fault, using the patch in
* https://issues.apache.org/jira/secure/attachment/12515544/AVRO-1034.patch
*
@@ -281,11 +281,12 @@ int add_array( avro_writer_t writer,
try ( avro_value_write( writer, _record ),
"Unable to write record into memory using writer_record" );
-
+
print_array_fields( _record );
avro_value_decref( _record );
avro_value_iface_decref( writer_class );
+avro_value_iface_decref(
[jira] [Commented] (AVRO-2026) Incorporate valgrind memory testing for C
[ https://issues.apache.org/jira/browse/AVRO-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715405#comment-16715405 ] ASF subversion and git services commented on AVRO-2026: --- Commit 7fbedbb8923bb203409f5d5576a38ac70aba9f4b in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=7fbedbb ] [AVRO-2026] Add valgrind to c tests This closes #218 Squashed merge of: Don't clean after test. Allow review of memcheck files Add valgrind to tests Correct memory leaks in tests > Incorporate valgrind memory testing for C > - > > Key: AVRO-2026 > URL: https://issues.apache.org/jira/browse/AVRO-2026 > Project: Apache Avro > Issue Type: Improvement > Components: c >Reporter: John Gill >Priority: Major > Fix For: 1.8.3 > > Attachments: AVRO-2026.patch > > > With the work done for AVRO-1167 & AVRO-766 in identifying memory leaks > decided to make it a standard part of the testing framework. This issue > requires https://github.com/apache/avro/pull/218 otherwise test_avro_schema > and test_avro_values fail due to to avro_schema_copy calls. > Adding valgrind tool to the docker container > Adding valgrind to CMakeLists.txt to optionally do valgrind testing if the > tool is found > Cleaning up existing tests that were failing due to not properly cleaning up > memory > Lastly, as part of ./build.sh test for lang/c do not remove the build/c > directory. The memcheck output is kept so that memory leak issues can be > reviewed and addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2238) Update Docker image from java to openjdk
[ https://issues.apache.org/jira/browse/AVRO-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715355#comment-16715355 ] ASF subversion and git services commented on AVRO-2238: --- Commit b4597aff0da39f58a018f92c4bdf3f9374c49e07 in avro's branch refs/heads/AVRO-2238 from [~Fokko] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b4597af ] AVRO-2238 Update Dockerfile base image from java to openjdk > Update Docker image from java to openjdk > > > Key: AVRO-2238 > URL: https://issues.apache.org/jira/browse/AVRO-2238 > Project: Apache Avro > Issue Type: Improvement > Components: docker >Reporter: Fokko Driesprong >Priority: Major > > Currently the docker image to run the tests is still using java which is > deprecated: https://hub.docker.com/_/java/ > Therefore we should move to openjdk (https://hub.docker.com/_/openjdk/). > Starting with version 8, and also adding 10 and 11 to it to make sure that > Avro is compatible with future version of Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715360#comment-16715360 ] ASF subversion and git services commented on AVRO-1167: --- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715364#comment-16715364 ] ASF subversion and git services commented on AVRO-1167: --- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715363#comment-16715363 ] ASF subversion and git services commented on AVRO-1167: --- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[
https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715357#comment-16715357
]
ASF GitHub Bot commented on AVRO-1167:
--
dkulp closed pull request #217: AVRO-1167, AVRO-766: Fix c AVRO_LINK memory
leaks
URL: https://github.com/apache/avro/pull/217
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/c/src/schema.c b/lang/c/src/schema.c
index 3ade1140e..97e3ff354 100644
--- a/lang/c/src/schema.c
+++ b/lang/c/src/schema.c
@@ -2,17 +2,17 @@
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
- * The ASF licenses this file to you under the Apache License, Version 2.0
+ * The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
- *
+ *
* http://www.apache.org/licenses/LICENSE-2.0
- *
+ *
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
* implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ * permissions and limitations under the License.
*/
#include "avro/allocation.h"
@@ -61,7 +61,7 @@ static int is_avro_id(const char *name)
}
}
/*
-* starts with [A-Za-z_] subsequent [A-Za-z0-9_]
+* starts with [A-Za-z_] subsequent [A-Za-z0-9_]
*/
return 1;
}
@@ -199,7 +199,13 @@ static void avro_schema_free(avro_schema_t schema)
case AVRO_LINK:{
struct avro_link_schema_t *link;
link = avro_schema_to_link(schema);
- avro_schema_decref(link->to);
+ /* Since we didn't increment the
+* reference count of the target
+* schema when we created the link, we
+* should not decrement the reference
+* count of the target schema when we
+* free the link.
+*/
avro_freet(struct avro_link_schema_t, link);
}
break;
@@ -727,7 +733,19 @@ avro_schema_t avro_schema_link(avro_schema_t to)
avro_set_error("Cannot allocate new link schema");
return NULL;
}
- link->to = avro_schema_incref(to);
+
+ /* Do not increment the reference count of target schema
+* pointed to by the AVRO_LINK. AVRO_LINKs are only valid
+* internal to a schema. The target schema pointed to by a
+* link will be valid as long as the top-level schema is
+* valid. Similarly, the link will be valid as long as the
+* top-level schema is valid. Therefore the validity of the
+* link ensures the validity of its target, and we don't need
+* an additional reference count on the target. This mechanism
+* of an implied validity also breaks reference count cycles
+* for recursive schemas, which result in memory leaks.
+*/
+ link->to = to;
avro_schema_init(>obj, AVRO_LINK);
return >obj;
}
@@ -807,7 +825,7 @@ avro_type_from_json_t(json_t *json, avro_type_t *type,
return EINVAL;
}
/*
-* TODO: gperf/re2c this
+* TODO: gperf/re2c this
*/
if (strcmp(type_str, "string") == 0) {
*type = AVRO_STRING;
@@ -1259,7 +1277,7 @@ avro_schema_from_json_length(const char *jsontext, size_t
length,
return avro_schema_from_json_root(root, schema);
}
-avro_schema_t avro_schema_copy(avro_schema_t schema)
+avro_schema_t avro_schema_copy_root(avro_schema_t schema, st_table
*named_schemas)
{
long i;
avro_schema_t new_schema = NULL;
@@ -1276,7 +1294,7 @@ avro_schema_t avro_schema_copy(avro_schema_t schema)
case AVRO_BOOLEAN:
case AVRO_NULL:
/*
-* No need to copy primitives since they're static
+* No need to copy primitives since they're static
*/
new_schema = schema;
break;
@@ -1288,6 +1306,10 @@ avro_schema_t
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715366#comment-16715366 ] ASF subversion and git services commented on AVRO-1167: --- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715367#comment-16715367 ] ASF subversion and git services commented on AVRO-766: -- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715368#comment-16715368 ] ASF subversion and git services commented on AVRO-766: -- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715358#comment-16715358 ] ASF subversion and git services commented on AVRO-1167: --- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715365#comment-16715365 ] ASF subversion and git services commented on AVRO-766: -- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715361#comment-16715361 ] ASF subversion and git services commented on AVRO-766: -- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1167) Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly.
[ https://issues.apache.org/jira/browse/AVRO-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715369#comment-16715369 ] ASF subversion and git services commented on AVRO-1167: --- Commit b597c3ab93ef1c6a37a9c1fa3584d86d20723784 in avro's branch refs/heads/master from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b597c3a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > Avro-C: avro_schema_copy() does not copy AVRO_LINKs properly. > - > > Key: AVRO-1167 > URL: https://issues.apache.org/jira/browse/AVRO-1167 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.1 > Environment: Ubuntu Linux 11.10 >Reporter: Vivek Nadkarni >Priority: Major > Attachments: AVRO-1167-TEST.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When avro_schema_copy() encounters an AVRO_LINK from an old_schema to a > new_schema, it sets the target of the new_link to the target of the old_link > in the old_schema. Thus, the AVRO_LINK in the new_schema points to an element > in the old_schema. > While this is currently safe, since the reference count of the target in the > old_schema is incremented, we are not really making a "copy" of the schema. > There is a "TODO" in the code, which says that we should make a > avro_schema_copy() of the target in old_schema instead of linking directly to > it. However, this solution of making a copy would result in a few problems: > 1. Avro schemas are intended to be self-contained. That implies that > AVRO_LINKs are intended to be internal links inside a self-contained schema. > The code introduces unnecessary (and potentially disallowed) external > dependencies in an Avro schema. > 2. The purpose of copying a schema that we want to decouple the old_schema > from the new_schema. The two copies may have different owners, we may want to > deallocate old schema etc. > 3. If the schema is recursive, then the code would enter an infinite > recursion loop. > It appears to me that the "correct" solution would be to replicate the entire > structure of the current schema, including the internal links. This means > that if old_link_A points to old_target_B, then new_link_A should point to > new_target_B in the new schema. Note that there should only be one copy of > new_target_B in the new schema, even if there are multiple links pointing to > new_target_B - i.e. we should not make a new copy for each link. > In order to implement this proper copying of links, we would need to keep a > lookup table of pairs of old and new schemas as they are being created, as > well as a list of all the AVRO_LINKs that are copied. Then as a post-copy > step, we would go and fix up all the AVRO_LINKs to point to the appropriate > targets. This is the way the schema is constructed in the first place in > avro_schema_from_json(). > An inefficient way to obtain the correct result from avro_schema_copy() would > be to perform an avro_schema_to_json() followed by an avro_schema_from_json(). > Note: I have not implemented a fix for this issue, but I am documenting this > issue in AVRO-JIRA because this issue needs to be resolved before AVRO-766 > can be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715362#comment-16715362 ] ASF subversion and git services commented on AVRO-766: -- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-766) C: Memory leak from reference count cycles
[ https://issues.apache.org/jira/browse/AVRO-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715359#comment-16715359 ] ASF subversion and git services commented on AVRO-766: -- Commit 1f00b1a7a2f76921325ba073de3212a4a22524de in avro's branch refs/heads/branch-1.8 from John Gill [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1f00b1a ] AVRO-1167, AVRO-766: Fix c AVRO_LINK memory leaks (#217) * AVRO-1167: Enhance avro_schema_copy for AVRO_LINK - Add hash of named schemas found during copy - Find saved named schema for copy of AVRO_LINK * AVRO-766: Correct memory leaks in AVRO_LINK copy - Adds test cases for AVRO-766 & AVRO-1167 - Corrects reference counting for avro_schema_copy * Enable TEST_AVRO_1167 in test_avro_766 This ensures that both fixes work together and that no valgrind errors are produced from a recrusive schema. > C: Memory leak from reference count cycles > -- > > Key: AVRO-766 > URL: https://issues.apache.org/jira/browse/AVRO-766 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.5.0 >Reporter: Douglas Creager >Priority: Major > Attachments: AVRO-766.patch, ref-cycle.c > > > If you parse a recursive Avro schema, you end up with a cycle in the > reference graph for the avro_schema_t objects that are created. The > reference counting mechanism that we're using can't detect this, and so you > get a memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1952) Prevent JIT compilation in C#
[ https://issues.apache.org/jira/browse/AVRO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715354#comment-16715354 ] ASF subversion and git services commented on AVRO-1952: --- Commit eaaf263372229f9aad9970f54767e043b1e104dc in avro's branch refs/heads/AVRO-2238 from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=eaaf263 ] [AVRO-1952] Squashed commit of the following: This closes #134 commit 0cf2081569f3244ae535710d46ea41e15e57bbcc Author: dolow Date: Wed Sep 28 20:10:17 2016 +0900 removed unused GetConstructor methods commit aad24a7b1bbeb39105b8b3ee9578dbc792d8cfea Author: dolow Date: Wed Sep 28 17:45:43 2016 +0900 replaced Reflection.Emit to simple instantiation for supporting AOT. > Prevent JIT compilation in C# > - > > Key: AVRO-1952 > URL: https://issues.apache.org/jira/browse/AVRO-1952 > Project: Apache Avro > Issue Type: Improvement > Components: csharp > Environment: Typical Unity (C#) development for mobile, iOS > compilation. >Reporter: Yuki Kuwabara >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > C# code includes JIT API (Reflection.Emit) and it occurs error in iOS or > other platform that does not support JIT compile. > Reflection.Emit is used in very specific point and it can be replaced with > different safe code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2156) Allow custom namespace for C# generated code
[ https://issues.apache.org/jira/browse/AVRO-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715353#comment-16715353 ] ASF subversion and git services commented on AVRO-2156: --- Commit 602a2c5b5258d5e1f0f60f71506b16801cbdf156 in avro's branch refs/heads/AVRO-2238 from [~blachniet] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=602a2c5 ] AVRO-2156: Map Avro namespace to C# namespaces during code generation > Allow custom namespace for C# generated code > > > Key: AVRO-2156 > URL: https://issues.apache.org/jira/browse/AVRO-2156 > Project: Apache Avro > Issue Type: New Feature > Components: csharp >Affects Versions: 1.8.2 >Reporter: Brian Lachniet >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The namespace used in Avro schemas/protocols does not always map nicely to a > standard C# namespace. Avrogen (the C# code generator) currently uses the > Avro namespace as the C# namespace. I propose that we update the avrogen > utility to allow the user to override the namespace used in the generated > code via a command line parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1956) C sharp build fails in docker image
[
https://issues.apache.org/jira/browse/AVRO-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1956.
---
Resolution: Cannot Reproduce
Marking as not reproducible as with the latest Docker installs/versions and
latest "master" code, it seems to work fine.
> C sharp build fails in docker image
> ---
>
> Key: AVRO-1956
> URL: https://issues.apache.org/jira/browse/AVRO-1956
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>
> When you run the build in the centos docker image you get
> {quote}
> Unhandled Exception:
> System.ArgumentException: NUnit components for version 2.0.50727 of the CLR
> are not installed
> {quote}
> As noted by [~Simon24601] in the mailing list:
> {quote}
> From discussion with [~jcustenborder], it seems like the C# tests won't run
> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
> been done for 1.8.2) makes the tests run. But removing it makes the tests
> fail on my system (CentOS 7).
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1955) avrogen.exe supports only windows platform
[
https://issues.apache.org/jira/browse/AVRO-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715226#comment-16715226
]
ASF GitHub Bot commented on AVRO-1955:
--
dkulp closed pull request #137: AVRO-1955: fix avrogen.exe output dir path
format
URL: https://github.com/apache/avro/pull/137
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/csharp/src/apache/main/CodeGen/CodeGen.cs
b/lang/csharp/src/apache/main/CodeGen/CodeGen.cs
index aa31920c4..c8081fd6a 100644
--- a/lang/csharp/src/apache/main/CodeGen/CodeGen.cs
+++ b/lang/csharp/src/apache/main/CodeGen/CodeGen.cs
@@ -828,7 +828,11 @@ public virtual void WriteTypes(string outputdir)
{
var ns = nsc[i];
-string dir = outputdir + "\\" +
CodeGenUtil.Instance.UnMangle(ns.Name).Replace('.', '\\');
+string dir = outputdir;
+foreach (string name in
CodeGenUtil.Instance.UnMangle(ns.Name).Split('.'))
+{
+dir = Path.Combine(dir, name);
+}
Directory.CreateDirectory(dir);
var new_ns = new CodeNamespace(ns.Name);
@@ -840,7 +844,7 @@ public virtual void WriteTypes(string outputdir)
for (int j = 0; j < types.Count; j++)
{
var ctd = types[j];
-string file = dir + "\\" +
CodeGenUtil.Instance.UnMangle(ctd.Name) + ".cs";
+string file = Path.Combine(dir,
Path.ChangeExtension(CodeGenUtil.Instance.UnMangle(ctd.Name), "cs"));
using (var writer = new StreamWriter(file, false))
{
new_ns.Types.Add(ctd);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> avrogen.exe supports only windows platform
> --
>
> Key: AVRO-1955
> URL: https://issues.apache.org/jira/browse/AVRO-1955
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Reporter: Naruto TAKAHASHI
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> avrogen.exe output directory path separator is '\' .
> When executing avrogen.exe on macOS or Linux, output directory path is not
> expected.
> e.g. sample.avsc
> {code:javascript}
> {
> "type": "record",
> "name": "ExampleModel",
> "namespace": "com.example",
> "fields": [
> {
> "name": "num",
> "type": "long"
> },
> {
> "name": "str",
> "type": "string"
> }
> ]
> }
> {code}
> {code}
> mono ./build/codegen/Release/avrogen.exe -s sample.avsc output
> {code}
> avrogen.exe outputs below two regular files.
> - "output\com\example"
> - "output\com\example\ExampleModel.cs"
> avrogen.exe should use Path API of .NET when creating output directory path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1956) C sharp build fails in docker image
[
https://issues.apache.org/jira/browse/AVRO-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715332#comment-16715332
]
ASF GitHub Bot commented on AVRO-1956:
--
dkulp commented on issue #153: AVRO-1956 Workaround incompatibility in testing
CSharp code
URL: https://github.com/apache/avro/pull/153#issuecomment-445924523
Closing this as I don't believe it's needed anymore. I've created new
images (as a new Avro committer and also new Docker installs on new Mac) and
the tests all pass. Thus, I believe this is not longer required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C sharp build fails in docker image
> ---
>
> Key: AVRO-1956
> URL: https://issues.apache.org/jira/browse/AVRO-1956
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>
> When you run the build in the centos docker image you get
> {quote}
> Unhandled Exception:
> System.ArgumentException: NUnit components for version 2.0.50727 of the CLR
> are not installed
> {quote}
> As noted by [~Simon24601] in the mailing list:
> {quote}
> From discussion with [~jcustenborder], it seems like the C# tests won't run
> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
> been done for 1.8.2) makes the tests run. But removing it makes the tests
> fail on my system (CentOS 7).
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1956) C sharp build fails in docker image
[
https://issues.apache.org/jira/browse/AVRO-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715333#comment-16715333
]
ASF GitHub Bot commented on AVRO-1956:
--
dkulp closed pull request #153: AVRO-1956 Workaround incompatibility in testing
CSharp code
URL: https://github.com/apache/avro/pull/153
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/csharp/build.sh b/lang/csharp/build.sh
index cb9f7fc89..a19753acc 100755
--- a/lang/csharp/build.sh
+++ b/lang/csharp/build.sh
@@ -30,7 +30,11 @@ case "$1" in
test)
xbuild
-nunit-console Avro.nunit
+if [[ ${INSIDE_AVRO_DOCKER} == 'Yes' ]]; then
+ nunit-console --framework=4.0 Avro.nunit
+else
+ nunit-console Avro.nunit
+fi
;;
perf)
diff --git a/share/docker/Dockerfile b/share/docker/Dockerfile
index 01760554d..0de1ef73d 100644
--- a/share/docker/Dockerfile
+++ b/share/docker/Dockerfile
@@ -21,6 +21,8 @@ FROM java:7-jdk
WORKDIR /root
+ENV INSIDE_AVRO_DOCKER Yes
+
# Add the repository for node.js 4.x
RUN curl -sL https://deb.nodesource.com/setup_4.x | bash -
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C sharp build fails in docker image
> ---
>
> Key: AVRO-1956
> URL: https://issues.apache.org/jira/browse/AVRO-1956
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>
> When you run the build in the centos docker image you get
> {quote}
> Unhandled Exception:
> System.ArgumentException: NUnit components for version 2.0.50727 of the CLR
> are not installed
> {quote}
> As noted by [~Simon24601] in the mailing list:
> {quote}
> From discussion with [~jcustenborder], it seems like the C# tests won't run
> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
> been done for 1.8.2) makes the tests run. But removing it makes the tests
> fail on my system (CentOS 7).
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1952) Prevent JIT compilation in C#
[ https://issues.apache.org/jira/browse/AVRO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1952. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Prevent JIT compilation in C# > - > > Key: AVRO-1952 > URL: https://issues.apache.org/jira/browse/AVRO-1952 > Project: Apache Avro > Issue Type: Improvement > Components: csharp > Environment: Typical Unity (C#) development for mobile, iOS > compilation. >Reporter: Yuki Kuwabara >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > C# code includes JIT API (Reflection.Emit) and it occurs error in iOS or > other platform that does not support JIT compile. > Reflection.Emit is used in very specific point and it can be replaced with > different safe code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1952) Prevent JIT compilation in C#
[ https://issues.apache.org/jira/browse/AVRO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715320#comment-16715320 ] ASF subversion and git services commented on AVRO-1952: --- Commit eaaf263372229f9aad9970f54767e043b1e104dc in avro's branch refs/heads/master from [~dkulp] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=eaaf263 ] [AVRO-1952] Squashed commit of the following: This closes #134 commit 0cf2081569f3244ae535710d46ea41e15e57bbcc Author: dolow Date: Wed Sep 28 20:10:17 2016 +0900 removed unused GetConstructor methods commit aad24a7b1bbeb39105b8b3ee9578dbc792d8cfea Author: dolow Date: Wed Sep 28 17:45:43 2016 +0900 replaced Reflection.Emit to simple instantiation for supporting AOT. > Prevent JIT compilation in C# > - > > Key: AVRO-1952 > URL: https://issues.apache.org/jira/browse/AVRO-1952 > Project: Apache Avro > Issue Type: Improvement > Components: csharp > Environment: Typical Unity (C#) development for mobile, iOS > compilation. >Reporter: Yuki Kuwabara >Priority: Major > > C# code includes JIT API (Reflection.Emit) and it occurs error in iOS or > other platform that does not support JIT compile. > Reflection.Emit is used in very specific point and it can be replaced with > different safe code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1952) Prevent JIT compilation in C#
[
https://issues.apache.org/jira/browse/AVRO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715321#comment-16715321
]
ASF GitHub Bot commented on AVRO-1952:
--
dkulp closed pull request #134: AVRO-1952: Prevent JIT compilation in C#
URL: https://github.com/apache/avro/pull/134
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/lang/csharp/src/apache/main/Specific/ObjectCreator.cs
b/lang/csharp/src/apache/main/Specific/ObjectCreator.cs
index 5435adbc2..d1daa951a 100644
--- a/lang/csharp/src/apache/main/Specific/ObjectCreator.cs
+++ b/lang/csharp/src/apache/main/Specific/ObjectCreator.cs
@@ -1,4 +1,4 @@
-/**
+/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -30,7 +30,7 @@ public sealed class ObjectCreator
public static ObjectCreator Instance { get { return instance; } }
///
-/// Static generic dictionary type used for creating new dictionary
instances
+/// Static generic dictionary type used for creating new dictionary
instances
///
private Type GenericMapType = typeof(Dictionary<,>);
@@ -43,14 +43,12 @@ public sealed class ObjectCreator
/// Static generic nullable type used for creating new nullable
instances
///
private Type GenericNullableType = typeof(Nullable<>);
-
+
private readonly Assembly execAssembly;
private readonly Assembly entryAssembly;
private readonly bool diffAssembly;
public delegate object CtorDelegate();
-private Type ctorType = typeof(CtorDelegate);
-Dictionary ctors;
private ObjectCreator()
{
@@ -61,8 +59,6 @@ private ObjectCreator()
GenericMapType = typeof(Dictionary<,>);
GenericListType = typeof(List<>);
-
-ctors = new Dictionary();
}
public struct NameCtorKey : IEquatable
@@ -110,7 +106,7 @@ public override int GetHashCode()
/// the object type to locate
/// whether or not to throw an error if the
type wasn't found
/// the object type, or null if not found
-private Type FindType(string name,bool throwError)
+private Type FindType(string name,bool throwError)
{
Type type;
@@ -149,7 +145,7 @@ private Type FindType(string name,bool throwError)
{
if (name == t.Name || name == t.FullName) type = t;
}
-
+
if (type != null)
break;
}
@@ -185,7 +181,7 @@ public Type GetType(Schema schema)
case Schema.Type.Double:
return typeof(double);
case Schema.Type.Bytes:
-return typeof(byte[]);
+return typeof(byte[]);
case Schema.Type.String:
return typeof(string);
case Schema.Type.Union:
@@ -207,7 +203,7 @@ public Type GetType(Schema schema)
itemType = GetType(s1);
}
-if (null != itemType )
+if (null != itemType )
{
if (itemType.IsValueType && !itemType.IsEnum)
{
@@ -217,7 +213,7 @@ public Type GetType(Schema schema)
}
catch (Exception) { }
}
-
+
return itemType;
}
}
@@ -272,28 +268,6 @@ public Type GetType(string name, Schema.Type schemaType)
return type;
}
-///
-/// Gets the default constructor for the specified type
-///
-/// name of object for the type
-/// schema type for the object
-/// type of the object
-/// Default constructor for the type
-public CtorDelegate GetConstructor(string name, Schema.Type
schemaType, Type type)
-{
-ConstructorInfo ctorInfo = type.GetConstructor(Type.EmptyTypes);
-if (ctorInfo == null)
-throw new AvroException("Class " + name + " has no default
constructor");
-
-DynamicMethod dynMethod = new DynamicMethod("DM$OBJ_FACTORY_" +
name, typeof(object), null, type, true);
-ILGenerator ilGen = dynMethod.GetILGenerator();
-ilGen.Emit(OpCodes.Nop);
-ilGen.Emit(OpCodes.Newobj, ctorInfo);
-
[jira] [Resolved] (AVRO-1955) avrogen.exe supports only windows platform
[
https://issues.apache.org/jira/browse/AVRO-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1955.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> avrogen.exe supports only windows platform
> --
>
> Key: AVRO-1955
> URL: https://issues.apache.org/jira/browse/AVRO-1955
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Reporter: Naruto TAKAHASHI
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> avrogen.exe output directory path separator is '\' .
> When executing avrogen.exe on macOS or Linux, output directory path is not
> expected.
> e.g. sample.avsc
> {code:javascript}
> {
> "type": "record",
> "name": "ExampleModel",
> "namespace": "com.example",
> "fields": [
> {
> "name": "num",
> "type": "long"
> },
> {
> "name": "str",
> "type": "string"
> }
> ]
> }
> {code}
> {code}
> mono ./build/codegen/Release/avrogen.exe -s sample.avsc output
> {code}
> avrogen.exe outputs below two regular files.
> - "output\com\example"
> - "output\com\example\ExampleModel.cs"
> avrogen.exe should use Path API of .NET when creating output directory path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
