[jira] [Updated] (AVRO-1614) Always getting a value...
[
https://issues.apache.org/jira/browse/AVRO-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niels Basjes updated AVRO-1614:
---
Attachment: AVRO-1614-20141202-v3.patch
Updated the patch:
- Fixed a bug (cloning a Builder now clones recursively)
- Fixed existing unit test
- Added specific unit tests for new feature.
Always getting a value...
-
Key: AVRO-1614
URL: https://issues.apache.org/jira/browse/AVRO-1614
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Niels Basjes
Attachments: AVRO-1614-20141027-v1.patch,
AVRO-1614-20141201-v2.patch, AVRO-1614-20141202-v3.patch
Sometimes the Avro structure becomes deeply nested.
If in such a scenario you want to be able to set a specific value deep in
this tree you want to do this:
{code}
public void setSomething(String value) {
myStruct
.getFoo()
.getBar()
.getOne()
.getOther()
.setSomething(value);
}
{code}
The 'problem' I ran into is that any of the 4 get methods can return a null
value so the code I have to write is really huge.
For every step in this method I have to build null checks and create the
underlying instance if it is null.
I already started writing helper methods to do this for parts of my tree.
To solve this in a way that makes this code readable I came up with the
following which I want to propose to you guys (before I start working on a
patch).
My idea is to generate a new 'get' method in addition to the existing normal
get method for the regular instance of the class.
So in addition to the
{code}
public Foo getFoo() {
return foo;
}
{code}
I propose to generate something like this as well in the cases where this is
a type of structure that you may want to traverse as shown in the example.
{code}
public Foo getAlwaysFoo() {
if (foo == null) {
setFoo(Foo.newBuilder().build());
}
return foo;
}
{code}
This way the automatically created instance immediately has all the defaults
I have defined.
Assuming this naming my code will be readable because it will look like this:
{code}
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1614) Always getting a value...
[
https://issues.apache.org/jira/browse/AVRO-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niels Basjes updated AVRO-1614:
---
Labels: java (was: )
Status: Patch Available (was: Open)
Always getting a value...
-
Key: AVRO-1614
URL: https://issues.apache.org/jira/browse/AVRO-1614
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Niels Basjes
Labels: java
Attachments: AVRO-1614-20141027-v1.patch,
AVRO-1614-20141201-v2.patch, AVRO-1614-20141202-v3.patch
Sometimes the Avro structure becomes deeply nested.
If in such a scenario you want to be able to set a specific value deep in
this tree you want to do this:
{code}
public void setSomething(String value) {
myStruct
.getFoo()
.getBar()
.getOne()
.getOther()
.setSomething(value);
}
{code}
The 'problem' I ran into is that any of the 4 get methods can return a null
value so the code I have to write is really huge.
For every step in this method I have to build null checks and create the
underlying instance if it is null.
I already started writing helper methods to do this for parts of my tree.
To solve this in a way that makes this code readable I came up with the
following which I want to propose to you guys (before I start working on a
patch).
My idea is to generate a new 'get' method in addition to the existing normal
get method for the regular instance of the class.
So in addition to the
{code}
public Foo getFoo() {
return foo;
}
{code}
I propose to generate something like this as well in the cases where this is
a type of structure that you may want to traverse as shown in the example.
{code}
public Foo getAlwaysFoo() {
if (foo == null) {
setFoo(Foo.newBuilder().build());
}
return foo;
}
{code}
This way the automatically created instance immediately has all the defaults
I have defined.
Assuming this naming my code will be readable because it will look like this:
{code}
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Always getting a value...
I uploaded a patch for AVRO-1614 and AVRO-1616 (just an update to the
.gitignore).
It seems that there is no automatic pickup by Jenkins configured (like with
Hadoop and HBase).
How do I proceed in getting your feedback on these patches and getting them
reviewed and if possible comitted?
Thanks.
Niels Basjes
On Mon, Dec 1, 2014 at 6:05 PM, Niels Basjes ni...@basjes.nl wrote:
Hi,
I realized over the weekend that the Builder instances do support an
'incomplete' schema and this is validated at the time the build() is called.
Based upon that I redid the patch today so that in a Builder in addition
to the actual value of a field there is now also a Builder field for that
field possible.
If that is used then you can have the incomplete form of the sub-schema in
a Builder.
So for any Builder instance there is a getFooBuilder() that either returns
the existing or creates a new Builder instance for the Foo field if such a
builder is supported.
As a consequence:
- schema validation is postponed until the actual build() is called.
- for the fields where this Builder is used the actual build() call
becomes recursive.
So in my testing code I can now do this:
*Measurement.Builder measurementBuilder = Measurement.newBuilder();*
*measurementBuilder*
*.getTransportBuilder()*
* .getConnectionBuilder()*
*.getNetworkConnectionBuilder()*
* .setNetworkAddress(127.0.0.1)*
* .setNetworkType(NetworkType.IPv4);*
*Measurement measurement = measurementBuilder.build();*
Open question: I have not seen unit tests that validate the generated Java
code yet. How to approach this?
Niels Basjes
On Thu, Nov 27, 2014 at 5:37 PM, Niels Basjes ni...@basjes.nl wrote:
Hi,
To see how it would work I simply wrote the patch and played in my
environment with the effects on the generated Java code.
[ I created a Jira issue and attached this draft patch to it:
https://issues.apache.org/jira/browse/AVRO-1614 ]
The idea works but I also found that for my usecase it is not very
pleasant to work with.
Assume this example again:
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}
The main problem is that in order to do .getAlwaysOne() I MUST define
ALL fields of that type with a default value.
What I don;t like about that is that I want the schema definition to
enforce the fact that some fields are mandatory.
By adding a default value to 'everything' I lose that capability of AVRO
... which I don't want.
At this point in time the only workaround this (for me major) issue is by
introducing something where I can do something like having a 'tree of
incomplete Builders' and when I say 'build()' to the top one it will build
the entire tree.
Any thoughts?
Do you guy think there is a need/usecase for this idea (so I leave the
issue open) or shall I simply close AVRO-1614 with a 'Won't fix'?
Niels Basjes
On Thu, Nov 27, 2014 at 1:37 AM, Doug Cutting cutt...@apache.org wrote:
On Wed, Nov 26, 2014 at 2:33 PM, svante karlsson s...@csi.se wrote:
I'm not sure that works for avro where null is used for an optional
field.
It should work. If it doesn't, it's a bug. For example:
record Foo {
union {string, null} name = default;
}
record Bar {
union {Foo, null} foo = {name = non-default};
}
Default values in IDL are JSON format.
Doug
--
Best regards / Met vriendelijke groeten,
Niels Basjes
--
Best regards / Met vriendelijke groeten,
Niels Basjes
--
Best regards / Met vriendelijke groeten,
Niels Basjes
[jira] [Commented] (AVRO-1614) Always getting a value...
[
https://issues.apache.org/jira/browse/AVRO-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231861#comment-14231861
]
Doug Cutting commented on AVRO-1614:
I like the builder-based approach much better than the original proposal.
It would be good to add a test with a field named builder and/or Builder to
make sure that names are properly escaped.
There are a lot of whitespace changes in the patch that distract from the
intended changes. Can you please generate a patch without these?
As for tests, there are two TestSpecificCompiler classes, one in
lang/java/compiler and one in lang/java/ipc. The latter validates that
generated code can be compiled by the Java compiler.
TestSpecificCompilerTool.java also validates specific compiler output. A
number of other tests depend on generated classes (TestSpecificData,
TestProtocolSpecific, etc.).
Always getting a value...
-
Key: AVRO-1614
URL: https://issues.apache.org/jira/browse/AVRO-1614
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Niels Basjes
Labels: java
Attachments: AVRO-1614-20141027-v1.patch,
AVRO-1614-20141201-v2.patch, AVRO-1614-20141202-v3.patch
Sometimes the Avro structure becomes deeply nested.
If in such a scenario you want to be able to set a specific value deep in
this tree you want to do this:
{code}
public void setSomething(String value) {
myStruct
.getFoo()
.getBar()
.getOne()
.getOther()
.setSomething(value);
}
{code}
The 'problem' I ran into is that any of the 4 get methods can return a null
value so the code I have to write is really huge.
For every step in this method I have to build null checks and create the
underlying instance if it is null.
I already started writing helper methods to do this for parts of my tree.
To solve this in a way that makes this code readable I came up with the
following which I want to propose to you guys (before I start working on a
patch).
My idea is to generate a new 'get' method in addition to the existing normal
get method for the regular instance of the class.
So in addition to the
{code}
public Foo getFoo() {
return foo;
}
{code}
I propose to generate something like this as well in the cases where this is
a type of structure that you may want to traverse as shown in the example.
{code}
public Foo getAlwaysFoo() {
if (foo == null) {
setFoo(Foo.newBuilder().build());
}
return foo;
}
{code}
This way the automatically created instance immediately has all the defaults
I have defined.
Assuming this naming my code will be readable because it will look like this:
{code}
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Deleted] (AVRO-1615) ciao
[ https://issues.apache.org/jira/browse/AVRO-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting deleted AVRO-1615: --- ciao Key: AVRO-1615 URL: https://issues.apache.org/jira/browse/AVRO-1615 Project: Avro Issue Type: Bug Reporter: emanuele -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: avro_generic_value_free function implemetation not defined
Hi all, Any updates on the nested record thing? Thanks, Vikas On Sun, Nov 30, 2014 at 11:31 AM, Vikas Saxena vikas.saxena.2...@gmail.com wrote: Thanks Doug, I am currently doing the same but the function avro_value_decref does not work as expected for nested records. I am attaching my test code where I have modified the PERSON_SCHEMA to chnage the record person to have a sub-field FullName which in turn is a record containing two field firstname and lastname. The problem is when I try to decref the avro variable for the sub-record, I get a segmentation fault. Attached is the source code for your reference. The issue occurs for line number 117 and 177. currently I have commented out those two lines. If this program runs 24x7 to parse constantly generated files, do u think there can be a possibility of a memory leak if line 117 and 177 and are left commented out? Thanks, Vikas On Sun, Nov 30, 2014 at 6:12 AM, Douglas Creager doug...@creagertino.net wrote: While invoking the avro_generic_value_free function, I get the error, implementation not defined. I looked through header file but couldn’t find the function. Is this a known issue? That looks like a bug in the documentation. Try avro_value_decref instead. cheers –doug -- Thanks and regards, Vikas Saxena. -- Thanks and regards, Vikas Saxena.
Re: avro_generic_value_free function implemetation not defined
Hi Vikas, To my understanding you only need to decref avro_value_t that you allocate with avro_generic_value_new. So you don’t need to decref values that you get from avro_value_get_by_name. You can run your example program with valgrind to verify this. -Mika On 03 Dec 2014, at 02:44, Vikas Saxena vikas.saxena.2...@gmail.com wrote: Hi all, Any updates on the nested record thing? Thanks, Vikas On Sun, Nov 30, 2014 at 11:31 AM, Vikas Saxena vikas.saxena.2...@gmail.com wrote: Thanks Doug, I am currently doing the same but the function avro_value_decref does not work as expected for nested records. I am attaching my test code where I have modified the PERSON_SCHEMA to chnage the record person to have a sub-field FullName which in turn is a record containing two field firstname and lastname. The problem is when I try to decref the avro variable for the sub-record, I get a segmentation fault. Attached is the source code for your reference. The issue occurs for line number 117 and 177. currently I have commented out those two lines. If this program runs 24x7 to parse constantly generated files, do u think there can be a possibility of a memory leak if line 117 and 177 and are left commented out? Thanks, Vikas On Sun, Nov 30, 2014 at 6:12 AM, Douglas Creager doug...@creagertino.net wrote: While invoking the avro_generic_value_free function, I get the error, implementation not defined. I looked through header file but couldn’t find the function. Is this a known issue? That looks like a bug in the documentation. Try avro_value_decref instead. cheers –doug -- Thanks and regards, Vikas Saxena. -- Thanks and regards, Vikas Saxena.
