C Library Avro Record Fields Question

[Gill, John]

Hello,
I am looking at using the Avro C Library API for a project and was wondering 
why Record Fields are not first class "objects"? Meaning, there is not an 
AVRO_FIELD type for them. Looking at the python and Java implementations there 
is a Field Class in both languages that can be accessed and utilized. However, 
in the C implementation there is an avro_schema_field_t but the index and name 
attributes are sort of useless (except for hashing) and the type is always 
returned. Additionally, all the interfaces for accessing a field are done via 
avro_schema_record_field_*.

I was looking to try and add some default value support for the project I am 
working on but am finding that without the ability to return an AVRO_FIELD type 
that the interfaces get ugly very quickly.

avro_value_t avro_schema_record_field_get_default_by_index(rSchema, i);
avro_value_t avro_schema_record_field_get_default_by_name(rSchema, 
"field1");
int avro_schema_record_field_set_default_by_index(rSchema, i, value);
int avro_schema_record_field_set_default_by_name(rSchema, "field1", value);

Additionally, if we wanted to add any other attributes from the Specification 
("doc", "order", "aliases") then the same very long API call names persist.

char * avro_schema_record_field_get_doc_by_index(rSchema, i);
char * avro_schema_record_field_get_doc_by_name(rSchema, "field1");
int avro_schema_record_field_set_doc_by_index(rSchema, i, value);
int avro_schema_record_field_set_doc_by_name(rSchema, "field1", value);

The reason I ask, and it's problematic, is because such a change would break 
existing API functionality. Currently a call to avro_schema_record_field_get 
returns the type attribute of an avro_record_field_t. With the proposed change 
the new avro_record_field_t would look like:

struct avro_record_field_t {
struct avro_obj_t obj;
int index;
char *name;
avro_schema_t type;
}
Thus, the obj attribute would be returned and a new set of API functions would 
be used to access information about a record field:

int avro_schema_field_get_index(fScehma);
char * avro_schema_field_get_name(fSchema);
avro_schema_t avro_schema_field_get_type(fSchema);
avro_value_t avro_schema_field_get_default(fSchema);
char * avro_schema_field_get_doc(fSchema);

Interested in thoughts about the change and possibly why it was designed this 
way. Looking at C++ it seems to have the same issue.

Thanks,
- John

[jira] [Updated] (AVRO-1723) Add support for forward declarations in avro IDL

[Zoltan Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Farkas updated AVRO-1723:

Release Note: 
Pull request available:

https://github.com/apache/avro/pull/79
  Status: Patch Available  (was: Open)

Pull request available:

https://github.com/apache/avro/pull/79

> Add support for forward declarations in avro IDL
> 
>
> Key: AVRO-1723
> URL: https://issues.apache.org/jira/browse/AVRO-1723
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
> Attachments: AVRO-1723.patch
>
>
> Currently Recursive data structures like:
> record SampleNode {
>int count = 0;
>array samples = [];
> }
> record SamplePair {
>  string name;
>  SampleNode node;
> }
> It is not possible to declare in IDL,
> however it is possible to declare in avsc (with fix from 
> https://issues.apache.org/jira/browse/AVRO-1667 )
> It is actually not complicated to implement, here is some detail on a 
> possible implementation:
> https://github.com/zolyfarkas/avro/commit/210c50105717149f3daa39b8d4160b8548b8e363
> This would close a capability gap with google protocol buffers...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (AVRO-1723) Add support for forward declarations in avro IDL

[Zoltan Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Farkas reassigned AVRO-1723:
---

Assignee: Zoltan Farkas

> Add support for forward declarations in avro IDL
> 
>
> Key: AVRO-1723
> URL: https://issues.apache.org/jira/browse/AVRO-1723
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
> Attachments: AVRO-1723.patch
>
>
> Currently Recursive data structures like:
> record SampleNode {
>int count = 0;
>array samples = [];
> }
> record SamplePair {
>  string name;
>  SampleNode node;
> }
> It is not possible to declare in IDL,
> however it is possible to declare in avsc (with fix from 
> https://issues.apache.org/jira/browse/AVRO-1667 )
> It is actually not complicated to implement, here is some detail on a 
> possible implementation:
> https://github.com/zolyfarkas/avro/commit/210c50105717149f3daa39b8d4160b8548b8e363
> This would close a capability gap with google protocol buffers...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Release Apache Avro 1.8.2 RC2

[Niels Basjes]

Hi,

NOTE: I'm at the Hadoop Summit in Munchen Germany with limited tools (and
limited power for my laptop) here.

I did a git clone of the avro code based and checked out the project at
commit
6bbcf7658e31cd7e92c459300fd70263b9b89de6

I found a simple problem that needs to be fixed:
Running ./build.sh rat reveals a build failure caused by a missing
copyright comment in
lang/java/avro/src/test/java/org/apache/avro/TestFixed.java

Niels Basjes




On Thu, Apr 6, 2017 at 12:16 AM, Harsha  wrote:

> Sean,
>  Agree on most of points made here. I am new to the avro project but
> also active committer/PMC in other apache projects.
> I understand that contributors can help getting the reviews up to the
> speed but at the end of the day patches need to be merged
> in to make it into the release.
> There are quite a few patches with reviews available but not much activity
> in-terms of merging or any further feedback. I am happy to help
> wherever I can. I’ll open a new thread for further discussion on this.
>
> To get back on track of voting.
>
> 1. Ran unit tests 1.8.2 rc tag
> 2. Verified signature of artifacts
> 3. Tested avro-java module
>
>
> +1 (non-binding).
>
>
> Thanks,
> Harsha
>
> On Apr 5, 2017, 2:29 PM -0700, Sean Busbey , wrote:
> > Personally, I think it's more important that we get back on our
> > footing for release cadence. There are already a ton of things in the
> > 1.8.2 RC. We can parallelize trying to get the other noted items
> > reviewed and then include them in a 1.8.3 release later (say with a
> > goal of next month).
> >
> > I don't want to hijack Suraj's VOTE thread, but the two things we most
> > need right now are 1) a release cadence we can use to set expectations
> > about when something that's been merged will see the light of day and
> > 2) more active reviewers. We're a volunteer run project, so our finite
> > supply of folks times has to be applied to both of these if we want to
> > succeed. If we hold up getting releases out for the work of recovering
> > our review bandwidth we'll perpetually be stuck with no releases.
> >
> > Due to the project's structure, only committers can help out with the
> > release machinery (and thanks a ton for taking this on Suraj!).
> > However, any contributor who has a spare hour here or there can help
> > out with reviews. For example, it takes much less effort for me as a
> > committer to go through patches once I see someone else has already
> > provided feedback, especially if it's someone who's been making a
> > habit of it.
> >
> > On Wed, Apr 5, 2017 at 2:09 PM, Harsha  wrote:
> > > Suraj,
> > > There are further comments on this JIRA https://issues.apache.org/
> jira/browse/AVRO-1885 to include additional patches in the release.
> > > It looks like the patch-merge process is really slow in Avro. But It
> would good to pick the patches raised by other users in the JIRA.
> > > Can we do a cut with those patches in.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Apr 5, 2017, 1:50 PM -0700, suraj acharya ,
> wrote:
> > > > +1 Non binding
> > > >
> > > >
> > > > -Suraj Acharya
> > > >
> > > > On Wed, Apr 5, 2017 at 3:49 PM, Suraj Acharya 
> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I propose the following RC to be released as official Apache Avro
> 1.8.2
> > > > > release.
> > > > >
> > > > > The commit id is 6bbcf7658e31cd7e92c459300fd70263b9b89de6
> > > > > * This corresponds to the tag: release-1.8.2-rc2
> > > > > * https://s.apache.org/avro-1.8.2-rc2
> > > > >
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc2/
> > > > >
> > > > > You can find the KEYS file here:
> > > > > * https://dist.apache.org/repos/dist/release/avro/KEYS
> > > > >
> > > > > Binary artifacts for Java are staged in Nexus here:
> > > > > * https://repository.apache.org/content/repositories/
> orgapacheavro-1010/
> > > > >
> > > > > This release includes:
> > > > > * A spec for single-message Avro encoding and a Java implementation
> > > > > * Java: Bug fixes for Java logical types
> > > > > * Java: Support for Decimal with specific classes
> > > > > * Ruby: A fix for compatibility when using snappy
> > > > > * Python 3: Updated to use the standard module-level logging
> pattern
> > > > > * C++: Support for Boost >= 1.59
> > > > > * And more bug fixes...
> > > > >
> > > > > Please download, verify, and test. This vote will remain open for
> at least
> > > > > 72 hours. Given sufficient votes, I would like to close it on or
> about 9AM
> > > > > PDT on Wednesday, 12 April 2017.
> > > > >
> > > > > [ ] +1 Release this as Apache Avro 1.8.2
> > > > > [ ] +0
> > > > > [ ] -1 Do not release this because...
> > > > >
> > > > >
> > > > > --
> > > > > Suraj Acharya
> > > > >
> >
> >
> >
> > --
> > busbey
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Commented] (AVRO-1837) Add support for logical date, time, timestamp type for python AVRO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958349#comment-15958349
 ] 

ASF GitHub Bot commented on AVRO-1837:
--

GitHub user abrarsheikh opened a pull request:

https://github.com/apache/avro/pull/207

[AVRO-1837] adding support for date, time and timestamp logicaltype type 
support



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abrarsheikh/avro time_logical_type

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #207


commit cebef027e2255763dc15168d4624db5a5f24e9a7
Author: Abrar Sheikh 
Date:   2017-04-06T05:08:44Z

date, time and timestamp logictype type support




> Add support for logical date, time, timestamp type for python AVRO
> --
>
> Key: AVRO-1837
> URL: https://issues.apache.org/jira/browse/AVRO-1837
> Project: Avro
>  Issue Type: New Feature
>  Components: python
>Affects Versions: 1.9.0
>Reporter: Abrar Ahmed Sheikh
>
> Currently, python Avro does not seem to support a TIME, TIMESTAMP and DATE 
> logicalType type. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] avro pull request #207: [AVRO-1837] adding support for date, time and timest...

[abrarsheikh]

GitHub user abrarsheikh opened a pull request:

https://github.com/apache/avro/pull/207

[AVRO-1837] adding support for date, time and timestamp logicaltype type 
support



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abrarsheikh/avro time_logical_type

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #207


commit cebef027e2255763dc15168d4624db5a5f24e9a7
Author: Abrar Sheikh 
Date:   2017-04-06T05:08:44Z

date, time and timestamp logictype type support




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---