[jira] [Commented] (AVRO-1723) Add support for forward declarations in avro IDL
[
https://issues.apache.org/jira/browse/AVRO-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990573#comment-15990573
]
Thiruvalluvan M. G. commented on AVRO-1723:
---
[~zolyfarkas] Great. I just added a test for forward reference:
https://github.com/zolyfarkas/avro/pull/2. Please merge it to your branch when
you can.
I'll pull your patch in two days.
> Add support for forward declarations in avro IDL
>
>
> Key: AVRO-1723
> URL: https://issues.apache.org/jira/browse/AVRO-1723
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
> Attachments: AVRO-1723.patch
>
>
> Currently Recursive data structures like:
> record SampleNode {
>int count = 0;
>array samples = [];
> }
> record SamplePair {
> string name;
> SampleNode node;
> }
> It is not possible to declare in IDL,
> however it is possible to declare in avsc (with fix from
> https://issues.apache.org/jira/browse/AVRO-1667 )
> It is actually not complicated to implement, here is some detail on a
> possible implementation:
> https://github.com/zolyfarkas/avro/commit/210c50105717149f3daa39b8d4160b8548b8e363
> This would close a capability gap with google protocol buffers...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-2023) backport and update license fixes for branch-1.7
[
https://issues.apache.org/jira/browse/AVRO-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990521#comment-15990521
]
Suraj Acharya commented on AVRO-2023:
-
Apart from the above mentioned files AVRO-2023 branch now fails on only 5 files
{code}
5 Unknown Licenses
***
Unapproved licenses:
lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/g.bar.js
lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/protovis-r3.2.js
lang/java/tools/src/main/resources/META-INF/cddl-1.0.text
lang/java/tools/src/main/resources/META-INF/cddl-1.1.text
lang/java/tools/src/main/resources/META-INF/mpl-2.0.text
***
{code}
If the above questions are answered, I will merge
https://github.com/spacharya/avro/commits/AVRO-2023 to branch-1.7 and continue
on the release process.
> backport and update license fixes for branch-1.7
>
>
> Key: AVRO-2023
> URL: https://issues.apache.org/jira/browse/AVRO-2023
> Project: Avro
> Issue Type: Task
> Components: build, community
>Affects Versions: 1.7.7
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.7.8
>
>
> Backport the changes from AVRO-1722 and subtasks, ensure they're applicable
> for branch-1.7.
> git history as summary:
> {code}
> $ git hist --grep AVRO-1722 --grep AVRO-1727 --grep AVRO-1728 --grep
> AVRO-1729 --grep AVRO-1730 --grep AVRO-1731 --grep AVRO-1732 --grep AVRO-1733
> --grep AVRO-1734 --grep AVRO-1735 --grep AVRO-1736 --grep AVRO-1771
> origin/branch-1.8
> * 059e6df - AVRO-1771. Add LICENSE and NOTICE to avro-doc artifact.
> Contributed by blue. (1 year, 3 months ago)
> * ee28a20 - AVRO-1728: Java: Add LICENSE and NOTICE files to jars. (1 year, 4
> months ago)
> * 80ba788 - AVRO-1722 ADDENDUM: Add last license doc changes, rat helper. (1
> year, 4 months ago)
> * bf751df - AVRO-1722 ADDENDUM: Java: Fix tests broken by adding licenses. (1
> year, 4 months ago)
> * f49b2d3 - AVRO-1722: Update root LICENSE.txt and NOTICE.txt. (1 year, 4
> months ago)
> * 2c6c570 - AVRO-1730: Python3: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * e11c588 - AVRO-1730: Python: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * ada69ac - AVRO-1736: PHP: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * 96d2d35 - AVRO-1735: Perl: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * df5c513 - AVRO-1731: C: Add LICENSE and NOTICE to binary distribution. (1
> year, 4 months ago)
> * 3585c45 - AVRO-1733: C#: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * 5cb83db - AVRO-1732: C++: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * bc19e3a - AVRO-1729: Ruby: Add LICENSE and NOTICE to ruby gems. (1 year, 6
> months ago)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (AVRO-2023) backport and update license fixes for branch-1.7
[
https://issues.apache.org/jira/browse/AVRO-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990512#comment-15990512
]
Suraj Acharya edited comment on AVRO-2023 at 5/1/17 12:01 AM:
--
Some of the files has a different licensing :
*
https://github.com/apache/avro/blob/branch-1.7/lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/protovis-r3.2.js
Stanford Visualization Group has the license.
*
https://github.com/apache/avro/blob/branch-1.7/lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/g.bar.js
is licensed under the MIT license.
* In the changeset of AVRO-1728
https://github.com/spacharya/avro/blob/AVRO-2023/lang/java/tools/src/main/resources/META-INF/cddl-1.0.text
CDDL licensing was added. Do i just exclude the text files which have just the
licensing in them?
[~busbey]/ [~rdblue] is this okay?
was (Author: sacharya):
Some of the files has a different licensing :
*
https://github.com/apache/avro/blob/branch-1.7/lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/protovis-r3.2.js
Stanford Visualization Group has the license.
* In the changeset of AVRO-1728
https://github.com/spacharya/avro/blob/AVRO-2023/lang/java/tools/src/main/resources/META-INF/cddl-1.0.text
CDDL licensing was added. Do i just exclude the text files which have just the
licensing in them?
[~busbey]/ [~rdblue] is this okay?
> backport and update license fixes for branch-1.7
>
>
> Key: AVRO-2023
> URL: https://issues.apache.org/jira/browse/AVRO-2023
> Project: Avro
> Issue Type: Task
> Components: build, community
>Affects Versions: 1.7.7
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.7.8
>
>
> Backport the changes from AVRO-1722 and subtasks, ensure they're applicable
> for branch-1.7.
> git history as summary:
> {code}
> $ git hist --grep AVRO-1722 --grep AVRO-1727 --grep AVRO-1728 --grep
> AVRO-1729 --grep AVRO-1730 --grep AVRO-1731 --grep AVRO-1732 --grep AVRO-1733
> --grep AVRO-1734 --grep AVRO-1735 --grep AVRO-1736 --grep AVRO-1771
> origin/branch-1.8
> * 059e6df - AVRO-1771. Add LICENSE and NOTICE to avro-doc artifact.
> Contributed by blue. (1 year, 3 months ago)
> * ee28a20 - AVRO-1728: Java: Add LICENSE and NOTICE files to jars. (1 year, 4
> months ago)
> * 80ba788 - AVRO-1722 ADDENDUM: Add last license doc changes, rat helper. (1
> year, 4 months ago)
> * bf751df - AVRO-1722 ADDENDUM: Java: Fix tests broken by adding licenses. (1
> year, 4 months ago)
> * f49b2d3 - AVRO-1722: Update root LICENSE.txt and NOTICE.txt. (1 year, 4
> months ago)
> * 2c6c570 - AVRO-1730: Python3: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * e11c588 - AVRO-1730: Python: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * ada69ac - AVRO-1736: PHP: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * 96d2d35 - AVRO-1735: Perl: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * df5c513 - AVRO-1731: C: Add LICENSE and NOTICE to binary distribution. (1
> year, 4 months ago)
> * 3585c45 - AVRO-1733: C#: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * 5cb83db - AVRO-1732: C++: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * bc19e3a - AVRO-1729: Ruby: Add LICENSE and NOTICE to ruby gems. (1 year, 6
> months ago)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-2023) backport and update license fixes for branch-1.7
[
https://issues.apache.org/jira/browse/AVRO-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990512#comment-15990512
]
Suraj Acharya commented on AVRO-2023:
-
Some of the files has a different licensing :
*
https://github.com/apache/avro/blob/branch-1.7/lang/java/ipc/src/main/java/org/apache/avro/ipc/trace/static/protovis-r3.2.js
Stanford Visualization Group has the license.
* In the changeset of AVRO-1728
https://github.com/spacharya/avro/blob/AVRO-2023/lang/java/tools/src/main/resources/META-INF/cddl-1.0.text
CDDL licensing was added. Do i just exclude the text files which have just the
licensing in them?
[~busbey]/ [~rdblue] is this okay?
> backport and update license fixes for branch-1.7
>
>
> Key: AVRO-2023
> URL: https://issues.apache.org/jira/browse/AVRO-2023
> Project: Avro
> Issue Type: Task
> Components: build, community
>Affects Versions: 1.7.7
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.7.8
>
>
> Backport the changes from AVRO-1722 and subtasks, ensure they're applicable
> for branch-1.7.
> git history as summary:
> {code}
> $ git hist --grep AVRO-1722 --grep AVRO-1727 --grep AVRO-1728 --grep
> AVRO-1729 --grep AVRO-1730 --grep AVRO-1731 --grep AVRO-1732 --grep AVRO-1733
> --grep AVRO-1734 --grep AVRO-1735 --grep AVRO-1736 --grep AVRO-1771
> origin/branch-1.8
> * 059e6df - AVRO-1771. Add LICENSE and NOTICE to avro-doc artifact.
> Contributed by blue. (1 year, 3 months ago)
> * ee28a20 - AVRO-1728: Java: Add LICENSE and NOTICE files to jars. (1 year, 4
> months ago)
> * 80ba788 - AVRO-1722 ADDENDUM: Add last license doc changes, rat helper. (1
> year, 4 months ago)
> * bf751df - AVRO-1722 ADDENDUM: Java: Fix tests broken by adding licenses. (1
> year, 4 months ago)
> * f49b2d3 - AVRO-1722: Update root LICENSE.txt and NOTICE.txt. (1 year, 4
> months ago)
> * 2c6c570 - AVRO-1730: Python3: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * e11c588 - AVRO-1730: Python: Add LICENSE and NOTICE to distribution. (1
> year, 4 months ago)
> * ada69ac - AVRO-1736: PHP: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * 96d2d35 - AVRO-1735: Perl: Add LICENSE and NOTICE to distribution. (1 year,
> 4 months ago)
> * df5c513 - AVRO-1731: C: Add LICENSE and NOTICE to binary distribution. (1
> year, 4 months ago)
> * 3585c45 - AVRO-1733: C#: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * 5cb83db - AVRO-1732: C++: Add LICENSE and NOTICE to binary distribution. (1
> year, 6 months ago)
> * bc19e3a - AVRO-1729: Ruby: Add LICENSE and NOTICE to ruby gems. (1 year, 6
> months ago)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1723) Add support for forward declarations in avro IDL
[
https://issues.apache.org/jira/browse/AVRO-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990376#comment-15990376
]
Zoltan Farkas commented on AVRO-1723:
-
[~thiru_mg] Thanks for review.
your cleanup + fixes looks good.
Regarding the order of definition of named schema elements, it will not matter
anymore.
This implementation will resolve all named elements not only records.
I just tested the following:
{code}
/* Name Value record */
record ANameValue {
/** the name */
string name;
/** the value */
string value;
/* is the value a json object */
ValueType type = "PLAIN";
}
enum ValueType {JSON, BASE64BIN, PLAIN}
{code}
while this is not supported with AVRO without this change.
It will work just fine from now on.
Worthwhile adding a unit test.
What we should also do add a jira to move the generic schema walker we created
for this into the core avro lib and see if there are other places this can be
re-used as [~rdblue] suggested:
"Also, you might be able to simplify what's happening here by using the visitor
pattern. I've been thinking that it would be helpful to have an internal
SchemaVisitor for this kind of operation. That would cut down on the size of
these methods and separate the recursion from the logic you're trying to
implement. This may be a good time to add it to the core library."
This is the reason of existence of the CloningVisitor, to add one other use
case for this generic schema walker...
thank you!
> Add support for forward declarations in avro IDL
>
>
> Key: AVRO-1723
> URL: https://issues.apache.org/jira/browse/AVRO-1723
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
> Attachments: AVRO-1723.patch
>
>
> Currently Recursive data structures like:
> record SampleNode {
>int count = 0;
>array samples = [];
> }
> record SamplePair {
> string name;
> SampleNode node;
> }
> It is not possible to declare in IDL,
> however it is possible to declare in avsc (with fix from
> https://issues.apache.org/jira/browse/AVRO-1667 )
> It is actually not complicated to implement, here is some detail on a
> possible implementation:
> https://github.com/zolyfarkas/avro/commit/210c50105717149f3daa39b8d4160b8548b8e363
> This would close a capability gap with google protocol buffers...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1723) Add support for forward declarations in avro IDL
[
https://issues.apache.org/jira/browse/AVRO-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990340#comment-15990340
]
Thiruvalluvan M. G. commented on AVRO-1723:
---
[~zolyfarkas] I reviewed your work. It looks great. A few formatting issues and
conventions. There was one bug related to the use of Dequeue's {{getLast()}}.
Instead of asking you to make further changes (you have done several rounds
already), I made a pull request into your git repo.
https://github.com/zolyfarkas/avro/pull/1 Please see if my modifications are
good enough. If you like the changes, please merge them to your branch and
rebase it to the current master of Apache repo. If you don't like my changes or
want to modify parts of them, please go ahead.
I'll wait for a couple of days for others to comment. If there are no comments
or objections from the community, I'll merge your pull request.
Thank you for taking this up. This is a very useful improvement to the Schema
parser.
One more thing. You mention that the order of definition of named schema
elements in the IDL file doesn't matter any more. I think, while it is true for
Record types, the other named types such as Enum, Fixed must be defined at the
first occurrence. This is not a problem because they don't recurse. Probably in
a future patch we can take this restriction out as well.
> Add support for forward declarations in avro IDL
>
>
> Key: AVRO-1723
> URL: https://issues.apache.org/jira/browse/AVRO-1723
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
> Attachments: AVRO-1723.patch
>
>
> Currently Recursive data structures like:
> record SampleNode {
>int count = 0;
>array samples = [];
> }
> record SamplePair {
> string name;
> SampleNode node;
> }
> It is not possible to declare in IDL,
> however it is possible to declare in avsc (with fix from
> https://issues.apache.org/jira/browse/AVRO-1667 )
> It is actually not complicated to implement, here is some detail on a
> possible implementation:
> https://github.com/zolyfarkas/avro/commit/210c50105717149f3daa39b8d4160b8548b8e363
> This would close a capability gap with google protocol buffers...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1283) remove static caches from GenericData, SpecificData and ReflectData
[ https://issues.apache.org/jira/browse/AVRO-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990268#comment-15990268 ] Kenneth Knowles commented on AVRO-1283: --- I wanted to note that these caches also caused some trouble with classloaders in FLINK-5633 and BEAM-1970. Maybe this is known but since I didn't see that mentioned here, I thought I would add a comment. > remove static caches from GenericData, SpecificData and ReflectData > --- > > Key: AVRO-1283 > URL: https://issues.apache.org/jira/browse/AVRO-1283 > Project: Avro > Issue Type: Improvement >Reporter: Doug Cutting > > Currently GenericData, ReflectData and SpecificData are often treated as > singletons, through an instance returned by a static get() method. Because > of this, the caches they have must be thread-safe and are also memory leaks. > Increasingly of late, new instances of these are allocated for use by a given > DatumReader or DatumWriter. This issue explores whether we can: > - Change the static get() methods to always return new instances, so that > each DatumReader and DatumWriter has its own instance. Since DatumReader and > DatumWriter are not thread-safe, the thread safety requirement could then be > removed from caches, increasing their performance; and > - Change caches to be non-static, to remove potential memory leaks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
