[jira] [Updated] (AVRO-1663) C Library does not handle enum's namespace

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1663:
---
Attachment: AVRO-1663-4.patch

Thanks for reviewing, [~dcreager]! I've attached an updated (v4) patch that 
uses strrchr.

The v4 patch also leaves the function signature of {{avro_schema_fixed}} and 
{{avro_schema_enum}} unchanged, and instead adds two new functions  
{{avro_schema_fixed_ns}} and {{avro_schema_enum_ns}} that take an additional 
namespace parameter. This avoids a compatibility-breaking API change. It has 
the downside of making the API slightly inconsistent ({{avro_schema_record}} 
takes a namespace argument, but doesn't have a {{_ns}} suffix) but on balance 
this is probably the best compromise. What do you think?

Finally, could you confirm what changes to {{lang/c/version.sh}} are required? 
If I read the instructions correctly, it should be

{code}
libavro_micro_version++
libavro_interface_age=0
libavro_binary_age++
{code}

> C Library does not handle enum's namespace
> --
>
> Key: AVRO-1663
> URL: https://issues.apache.org/jira/browse/AVRO-1663
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Thomas Sanchez
>Assignee: Martin Kleppmann
> Attachments: AVRO-1663-2.patch, AVRO-1663-3.patch, AVRO-1663-4.patch, 
> AVRO-1663.patch
>
>
> {code}{
>   "type": "record",
>   "name": "EventName",
>   "namespace": "com.company.avro.schemas",
>   "fields": [
> {"name": "eventname_model",
>  "type": {
>"type": "enum",
>"namespace": "com.company.models",
>"name": "EventName",
>"symbols": [""]
>  }
> }
>  ]
> }
> {code}
> Such a schema is perfectly valid but the C library does no handle it because 
> it does not parse the namespace field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1617) C library: schema_record_equal can return true if schemas have different # of fields

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1617:
---
   Resolution: Fixed
Fix Version/s: 1.8.0
   Status: Resolved  (was: Patch Available)

I have committed this to branch-1.8 and trunk. Thanks for your contribution, 
[~skye]!

> C library: schema_record_equal can return true if schemas have different # of 
> fields
> 
>
> Key: AVRO-1617
> URL: https://issues.apache.org/jira/browse/AVRO-1617
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Skye Wanderman-Milne
>Assignee: Skye Wanderman-Milne
> Fix For: 1.8.0
>
> Attachments: AVRO-1617.patch
>
>
> schema_record_equal(a, b) will return true if schema b contains more fields 
> than schema a, as long as b contains all of a's fields. It should return 
> false in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1560) SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075129#comment-15075129
 ] 

ASF subversion and git services commented on AVRO-1560:
---

Commit 1722365 from [~martinkl] in branch 'avro/branches/branch-1.8'
[ https://svn.apache.org/r1722365 ]

AVRO-1560: C: Fix build for custom libsnappy location. Contributed by Ujjwal.

> SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt
> --
>
> Key: AVRO-1560
> URL: https://issues.apache.org/jira/browse/AVRO-1560
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Ujjwal
>Priority: Minor
>  Labels: build
> Attachments: AVRO-1560.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current CMakeLists.txt does not account for Snappy include folders 
> causing the compilation to fail when a custom snappy location is provided 
> using -DSNAPPY_ROOT_DIR
> e.g .
> {code:xml}
> cmake avro-c-1.7.7 -DSNAPPY_ROOT_DIR=${SNAPPY_INSTALL_DIR}
> {code}
> In 
> https://github.com/apache/avro/blob/release-1.7.7/lang/c/CMakeLists.txt#L143 
> we should include 
> {code:xml}
> include_directories(${SNAPPY_INCLUDE_DIR})
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1560) SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075130#comment-15075130
 ] 

ASF subversion and git services commented on AVRO-1560:
---

Commit 1722366 from [~martinkl] in branch 'avro/trunk'
[ https://svn.apache.org/r1722366 ]

AVRO-1560: C: Fix build for custom libsnappy location. Contributed by Ujjwal.

> SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt
> --
>
> Key: AVRO-1560
> URL: https://issues.apache.org/jira/browse/AVRO-1560
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Ujjwal
>Priority: Minor
>  Labels: build
> Attachments: AVRO-1560.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current CMakeLists.txt does not account for Snappy include folders 
> causing the compilation to fail when a custom snappy location is provided 
> using -DSNAPPY_ROOT_DIR
> e.g .
> {code:xml}
> cmake avro-c-1.7.7 -DSNAPPY_ROOT_DIR=${SNAPPY_INSTALL_DIR}
> {code}
> In 
> https://github.com/apache/avro/blob/release-1.7.7/lang/c/CMakeLists.txt#L143 
> we should include 
> {code:xml}
> include_directories(${SNAPPY_INCLUDE_DIR})
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1560) SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075214#comment-15075214
 ] 

Hudson commented on AVRO-1560:
--

SUCCESS: Integrated in AvroJava #563 (See 
[https://builds.apache.org/job/AvroJava/563/])
AVRO-1560: C: Fix build for custom libsnappy location. Contributed by Ujjwal. 
(martinkl: rev 1722366)
* trunk/CHANGES.txt
* trunk/lang/c/CMakeLists.txt


> SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt
> --
>
> Key: AVRO-1560
> URL: https://issues.apache.org/jira/browse/AVRO-1560
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Ujjwal
>Priority: Minor
>  Labels: build
> Fix For: 1.8.0
>
> Attachments: AVRO-1560.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current CMakeLists.txt does not account for Snappy include folders 
> causing the compilation to fail when a custom snappy location is provided 
> using -DSNAPPY_ROOT_DIR
> e.g .
> {code:xml}
> cmake avro-c-1.7.7 -DSNAPPY_ROOT_DIR=${SNAPPY_INSTALL_DIR}
> {code}
> In 
> https://github.com/apache/avro/blob/release-1.7.7/lang/c/CMakeLists.txt#L143 
> we should include 
> {code:xml}
> include_directories(${SNAPPY_INCLUDE_DIR})
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1617) C library: schema_record_equal can return true if schemas have different # of fields

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075213#comment-15075213
 ] 

Hudson commented on AVRO-1617:
--

SUCCESS: Integrated in AvroJava #563 (See 
[https://builds.apache.org/job/AvroJava/563/])
AVRO-1617: C: Fix equality checking of record schemas. Contributed by Skye 
Wanderman-Milne. (martinkl: rev 1722361)
* trunk/CHANGES.txt
* trunk/lang/c/src/schema_equal.c


> C library: schema_record_equal can return true if schemas have different # of 
> fields
> 
>
> Key: AVRO-1617
> URL: https://issues.apache.org/jira/browse/AVRO-1617
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Skye Wanderman-Milne
>Assignee: Skye Wanderman-Milne
> Fix For: 1.8.0
>
> Attachments: AVRO-1617.patch
>
>
> schema_record_equal(a, b) will return true if schema b contains more fields 
> than schema a, as long as b contains all of a's fields. It should return 
> false in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1572) Avro-C misses EOF on 4K boundary

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075212#comment-15075212
 ] 

Hudson commented on AVRO-1572:
--

SUCCESS: Integrated in AvroJava #563 (See 
[https://builds.apache.org/job/AvroJava/563/])
AVRO-1572: C: Fix EOF handling on data files that are multiples of 4096 bytes. 
Contributed by Ben Walsh. (martinkl: rev 1722359)
* trunk/CHANGES.txt
* trunk/lang/c/src/datafile.c
* trunk/lang/c/tests/CMakeLists.txt
* trunk/lang/c/tests/test_avro_1572.c


> Avro-C misses EOF on 4K boundary
> 
>
> Key: AVRO-1572
> URL: https://issues.apache.org/jira/browse/AVRO-1572
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Reporter: Ben Walsh
>Assignee: Martin Kleppmann
> Fix For: 1.8.0
>
> Attachments: AVRO-1572.patch
>
>
> As noted in AVRO-1405, Avro C is still missing EOF in the case that the sync 
> section ends exactly on a 4096 byte boundary (4096 bytes is the buffer size 
> used by the reader).
> I have fixed this bug (the suggested fix for AVRO-1364 actually works fine). 
> Also I added some more extensive testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1769) Depend on external libjansson instead of a bundled copy

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1769:
---
Attachment: AVRO-1769-2.patch.gz

[~dcreager]: Thanks for reviewing. I've attached an updated patch that uses 
pkgconfig, and removes the FindJansson.cmake file (and the reference to it in 
LICENSES). I've tested the build both on Mac OS (using Jansson from Homebrew) 
and in Docker.

> Depend on external libjansson instead of a bundled copy
> ---
>
> Key: AVRO-1769
> URL: https://issues.apache.org/jira/browse/AVRO-1769
> Project: Avro
>  Issue Type: Improvement
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Martin Kleppmann
>Assignee: Martin Kleppmann
> Attachments: AVRO-1769-2.patch.gz, AVRO-1769.patch.gz
>
>
> To fix AVRO-1691 we need to update to a newer version of Jansson. However, 
> rather than updating the version of Jansson that is bundled with Avro-C, this 
> would be a good time to switch to a dynamically linked version. As 
> [~edenhill] pointed out, libjansson is widely available in distributions 
> nowadays.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1725) Enum schema exhibits same restriction to enum symbols as to names

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1725:
---
Attachment: AVRO-1725.patch

Agree with [~cutting]. I've attached a documentation patch, adding the 
restriction on enum symbols to the spec.

> Enum schema exhibits same restriction to enum symbols as to names
> -
>
> Key: AVRO-1725
> URL: https://issues.apache.org/jira/browse/AVRO-1725
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.7
>Reporter: Nikita Makeev
> Attachments: AVRO-1725.patch
>
>
> EnumSchema class in org.apache.avro.Schema has the following code:
> for (String symbol : symbols)
> if (ordinals.put(validateName(symbol), i++) != null)
> which validates enum symbols using validateName() which makes impossible to 
> use symbols that are not conforming to standard for real names. 
> That prohibits using of symbols like "" (empty string) or anything starting 
> with number which does not seem to be intended.
> I guess this place requires either some another type of validation or no 
> validation at all. Can provide a patch for both cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1725) Enum schema exhibits same restriction to enum symbols as to names

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1725:
---
Status: Patch Available  (was: Open)

> Enum schema exhibits same restriction to enum symbols as to names
> -
>
> Key: AVRO-1725
> URL: https://issues.apache.org/jira/browse/AVRO-1725
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.7
>Reporter: Nikita Makeev
> Attachments: AVRO-1725.patch
>
>
> EnumSchema class in org.apache.avro.Schema has the following code:
> for (String symbol : symbols)
> if (ordinals.put(validateName(symbol), i++) != null)
> which validates enum symbols using validateName() which makes impossible to 
> use symbols that are not conforming to standard for real names. 
> That prohibits using of symbols like "" (empty string) or anything starting 
> with number which does not seem to be intended.
> I guess this place requires either some another type of validation or no 
> validation at all. Can provide a patch for both cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: AVRO-python #584

[Apache Jenkins Server]

See 

Changes:

[martinkl] AVRO-1560: C: Fix build for custom libsnappy location. Contributed 
by Ujjwal.

[martinkl] AVRO-1617: C: Fix equality checking of record schemas. Contributed 
by Skye Wanderman-Milne.

[martinkl] AVRO-1572: C: Fix EOF handling on data files that are multiples of 
4096 bytes. Contributed by Ben Walsh.

--
[...truncated 2023 lines...]
A lang/py/test/sample_http_server.py
A lang/py/test/test_io.py
A lang/py/test/test_datafile.py
A lang/py/test/test_datafile_interop.py
A lang/py/test/test_ipc.py
AUlang/py/test/test_tether_task.py
AUlang/py/test/mock_tether_parent.py
A lang/py/test/sample_http_client.py
A lang/py/test/txsample_http_server.py
A lang/py3
A lang/py3/avro
AUlang/py3/avro/datafile.py
A lang/py3/avro/NOTICE
A lang/py3/avro/LICENSE
AUlang/py3/avro/ipc.py
A lang/py3/avro/tests
AUlang/py3/avro/tests/av_bench.py
AUlang/py3/avro/tests/run_tests.py
AUlang/py3/avro/tests/sample_http_server.py
AUlang/py3/avro/tests/test_io.py
AUlang/py3/avro/tests/test_datafile.py
AUlang/py3/avro/tests/test_datafile_interop.py
AUlang/py3/avro/tests/test_ipc.py
AUlang/py3/avro/tests/txsample_http_client.py
AUlang/py3/avro/tests/sample_http_client.py
AUlang/py3/avro/tests/test_schema.py
AUlang/py3/avro/tests/test_protocol.py
AUlang/py3/avro/tests/gen_interop_data.py
AUlang/py3/avro/tests/txsample_http_server.py
A lang/py3/avro/tests/test_enum.py
AUlang/py3/avro/tests/test_script.py
AUlang/py3/avro/tool.py
AUlang/py3/avro/__init__.py
AUlang/py3/avro/schema.py
AUlang/py3/avro/protocol.py
AUlang/py3/avro/io.py
AUlang/py3/avro/txipc.py
A lang/py3/scripts
A lang/py3/scripts/avro
AUlang/py3/setup.py
A lang/py3/README.txt
A lang/js
AUlang/js/build.sh
A lang/js/etc
A lang/js/etc/browser
A lang/js/etc/browser/avro.js
A lang/js/etc/browser/crypto.js
A lang/js/etc/deprecated
AUlang/js/etc/deprecated/test_validator.js
AUlang/js/etc/deprecated/validator.js
A lang/js/etc/deprecated/Gruntfile.js
AUlang/js/etc/deprecated/README
A lang/js/README.md
A lang/js/test
A lang/js/test/dat
A lang/js/test/dat/person-10.no-codec.avro
A lang/js/test/dat/Person.avsc
A lang/js/test/dat/person-10.avro.raw
A lang/js/test/dat/Id.avsc
A lang/js/test/dat/person-10.avro
A lang/js/test/test_utils.js
A lang/js/test/test_files.js
A lang/js/test/test_schemas.js
A lang/js/NOTICE
A lang/js/LICENSE
A lang/js/package.json
A lang/js/doc
A lang/js/doc/Advanced-usage.md
A lang/js/doc/Home.md
A lang/js/doc/API.md
A lang/js/lib
A lang/js/lib/index.js
A lang/js/lib/utils.js
A lang/js/lib/files.js
A lang/js/lib/schemas.js
A share
A share/editors
A share/editors/avro-idl.vim
A share/editors/README.txt
A share/test
A share/test/interop
A share/test/interop/rpc
A share/test/interop/rpc/echo
A share/test/interop/rpc/echo/foo
AUshare/test/interop/rpc/echo/foo/request.avro
AUshare/test/interop/rpc/echo/foo/response.avro
A share/test/interop/rpc/hello
A share/test/interop/rpc/hello/world
AUshare/test/interop/rpc/hello/world/request.avro
AUshare/test/interop/rpc/hello/world/response.avro
A share/test/interop/rpc/add
A share/test/interop/rpc/add/onePlusOne
AUshare/test/interop/rpc/add/onePlusOne/request.avro
AUshare/test/interop/rpc/add/onePlusOne/response.avro
A share/test/interop/bin
AUshare/test/interop/bin/test_rpc_interop.sh
A share/test/schemas
A share/test/schemas/mail.avpr
A share/test/schemas/reserved.avsc
A share/test/schemas/specialtypes.avdl
A share/test/schemas/weather.avsc
A share/test/schemas/interop.avsc
A share/test/schemas/contexts.avdl
A share/test/schemas/BulkData.avpr
A share/test/schemas/social.avdl
A share/test/schemas/simple.avpr
A share/test/schemas/stringables.avdl
A share/test/schemas/echo.avdl
A share/test/schemas/namespace.avpr
A share/test/schemas/http.avdl
A share/test/schemas/FooBarSpecificRecord.avsc
A share/test/data
AUshare/test/data/weather-sorted.avro
AUshare/test/data/weather.avro
A share/test/data/weather.json
AUshare/test/data/test.avro12
AUshare/test/data/weather-snappy.avro
AU

[jira] [Resolved] (AVRO-1560) SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann resolved AVRO-1560.

   Resolution: Fixed
Fix Version/s: 1.8.0

+1. I needed this too in order to make the build work with Homebrew on Mac OS X.

I've committed this to branch-1.8 and trunk. Thanks for your contribution, 
[~ujjwal.wadhawan]!

> SNAPPY_INCLUDE_DIR not accounted in CMakeLists.txt
> --
>
> Key: AVRO-1560
> URL: https://issues.apache.org/jira/browse/AVRO-1560
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.7.7
>Reporter: Ujjwal
>Priority: Minor
>  Labels: build
> Fix For: 1.8.0
>
> Attachments: AVRO-1560.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The current CMakeLists.txt does not account for Snappy include folders 
> causing the compilation to fail when a custom snappy location is provided 
> using -DSNAPPY_ROOT_DIR
> e.g .
> {code:xml}
> cmake avro-c-1.7.7 -DSNAPPY_ROOT_DIR=${SNAPPY_INSTALL_DIR}
> {code}
> In 
> https://github.com/apache/avro/blob/release-1.7.7/lang/c/CMakeLists.txt#L143 
> we should include 
> {code:xml}
> include_directories(${SNAPPY_INCLUDE_DIR})
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Martin Kleppmann]

Yes, I think it's inevitable that there is some kind of registry or database of 
schemas, since a fingerprint by itself does not contain enough information to 
be able to decode a record. https://issues.apache.org/jira/browse/AVRO-1704 
calls this a SchemaStore.

In some cases, I imagine an application might be able to embed all schema 
versions in circulation into the application code. Then the SchemaStore could 
simply be an in-memory hash map. In this case, you'd have to be careful in 
deploying the application: when you want to add a new schema version, you first 
have to update all running instances to support the new schema (but not 
generate any records encoded with the new schema), and then in a second phase 
you update the application to start encoding records with the new schema.

In general, the SchemaStore might have to be a remote database, or something 
like the Confluent schema registry 
(http://docs.confluent.io/1.0/schema-registry/docs/intro.html). In that case, 
if an instance of the application encounters a schema fingerprint that it has 
not seen before, it can fetch the full schema from that database/registry.

The fingerprint could be a hash of the schema, bit it would also be ok to 
assign sequential numbers to schema versions, as long as you have some way of 
globally keeping track of those version numbers (this is what the Confluent 
schema registry does).

Martin

> On 23 Dec 2015, at 12:19, Niels Basjes  wrote:
> 
> Yes, as a generic crossplatform solution this makes a lot of sense. It s
> easy to build and stops consuming the messages as soon as the fingerprint
> changes.
> In my corporate reality I see that a source of messages puts them into
> Kafka, then several consumers read and deserialize them in near-realtime.
> In the static situation of "the schema doesn't change" having a fingerprint
> of the schema would suffice.
> But now the schema changes. I would like to deploy the new version of the
> source system, and consume these new messages using the schema migration
> idea. I.e. read the records as long as they are backwards compatible.
> As a general assumption: it is impossible to deploy a new version of all
> consuming applications at the same time.
> I want as little down time as possible and I do not want to lose any
> messages.
> 
> Is this possible with a simple fingerprint?
> The only direction I can think of right now is by having a central registry
> that contains all schemas ever used, indexed by their fingerprint.
> 
> What alternative solutions can we think of?
> 
> Niels
> 
> On Wed, 23 Dec 2015, 01:42 Sean Busbey  wrote:
> 
>> Including a schema fingerprint at the start
>> 
>> 1) reuses stuff we have
>> 2) gives a language independent notion of compatibility
>> 3) doesn't bind how folks get stuff in/out of the single record form.
>> 
>> --
>> Sean Busbey
>> On Dec 22, 2015 06:52, "Niels Basjes"  wrote:
>> 
>>> I was not clear enough in my previous email.
>>> What I meant is to 'wrap' the application schema in a serialization
>> wrapper
>>> schema that has a field indicating the "schema classname".
>>> That (generic setup) combined with some generated code in the schema
>>> classes should yield a solution that supports schema migration.
>>> 
>>> Niels
>>> 
>>> On Tue, Dec 22, 2015 at 11:55 AM, Niels Basjes  wrote:
>>> 
 Thanks for pointing this out.
 This is exactly what I was working on.
 
 The way I solved the 'does the schema match' question at work is by
 requiring that all schema's start with a single text field "schema
 classname" being the full class name of the class that was used to
>>> generate
 it.
 That way we can have newer versions of the schema and still be able to
 unpack them. In this form the classname is essentially an indicator if
 schema migration is possible; even though the schemas are different.
 
 What do you think of this direction?
 
 Niels
 
 
 On Mon, Dec 21, 2015 at 11:30 PM, Ryan Blue  wrote:
 
> Niels,
> 
> This sounds like a good idea to me to have methods like this. I've had
>>> to
> write those methods several times!
> 
> The idea is also related to AVRO-1704 [1], which is a suggestion to
> standardize the encoding that is used for single records. Some
>> projects
> have been embedding the schema fingerprint at the start of each
>> record,
>>> for
> example, which would be a helpful thing to do.
> 
> It may also be a good idea to create a helper object rather than
> attaching new methods to the datum classes themselves. In your example
> below, you have to create a new encoder or decoder for each method
>>> call. We
> could instead keep a backing buffer and encoder/decoder on a class
>> that
>>> the
> caller instantiates so that they can be reused. At the same time, that
> would make it possible to 

[jira] [Created] (AVRO-1776) update Ruby gem

[David Judd (JIRA)]

David Judd created AVRO-1776:


 Summary: update Ruby gem
 Key: AVRO-1776
 URL: https://issues.apache.org/jira/browse/AVRO-1776
 Project: Avro
  Issue Type: New Feature
  Components: ruby
Reporter: David Judd
Priority: Minor


The Ruby gem on rubygems.org is from 2014 and does not include, for example, 
the schema normalization features which were merged four months ago: 
https://rubygems.org/gems/avro/versions/1.7.7
https://github.com/apache/avro/commit/e0096f2f9d0036e02f16eb0b111fd68470758351



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1772) Ruby: load error in rubygem when loading schema normalization

[Martin Kleppmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074970#comment-15074970
 ] 

Martin Kleppmann commented on AVRO-1772:


+1. I just noticed two more files that are in the repository but not the 
manifest:

* test/test_fingerprints.rb
* test/test_schema_normalization.rb

Less critical since they're tests, but might as well add them since there's no 
good reason to leave them out.

> Ruby: load error in rubygem when loading schema normalization
> -
>
> Key: AVRO-1772
> URL: https://issues.apache.org/jira/browse/AVRO-1772
> Project: Avro
>  Issue Type: Bug
>  Components: ruby
>Affects Versions: 1.8.0
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.0
>
> Attachments: AVRO-1772.1.patch
>
>
> [~martinkl] found this bug while validating the 1.8.0 release:
> {code}
> kernel_require.rb:54:in `require': cannot load such file -- 
> avro/schema_normalization (LoadError)
> {code}
> {quote}
> It seems that the following two files, which are present in the source, are 
> missing from the gem file:
> lang/ruby/lib/avro/schema_normalization.rb
> lang/ruby/test/case_finder.rb
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1775) Running unit tests on Ruby 2.2

[Martin Kleppmann (JIRA)]

Martin Kleppmann created AVRO-1775:
--

 Summary: Running unit tests on Ruby 2.2
 Key: AVRO-1775
 URL: https://issues.apache.org/jira/browse/AVRO-1775
 Project: Avro
  Issue Type: Bug
  Components: ruby
Reporter: Martin Kleppmann


Ruby 2.2 [removed the test/unit framework from the standard 
library|https://bugs.ruby-lang.org/issues/9711#note-12]. As the Avro Ruby 
implementation uses it for its tests, we need to add a dependency on the 
{{test-unit}} gem in order to run the tests in Ruby 2.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1772) Ruby: load error in rubygem when loading schema normalization

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1772:
---
Attachment: AVRO-1772.2.patch

Missed one more file: test/test_schema.rb. Attached updated patch.

> Ruby: load error in rubygem when loading schema normalization
> -
>
> Key: AVRO-1772
> URL: https://issues.apache.org/jira/browse/AVRO-1772
> Project: Avro
>  Issue Type: Bug
>  Components: ruby
>Affects Versions: 1.8.0
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.0
>
> Attachments: AVRO-1772.1.patch, AVRO-1772.2.patch
>
>
> [~martinkl] found this bug while validating the 1.8.0 release:
> {code}
> kernel_require.rb:54:in `require': cannot load such file -- 
> avro/schema_normalization (LoadError)
> {code}
> {quote}
> It seems that the following two files, which are present in the source, are 
> missing from the gem file:
> lang/ruby/lib/avro/schema_normalization.rb
> lang/ruby/test/case_finder.rb
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1775) Running unit tests on Ruby 2.2

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1775:
---
Status: Patch Available  (was: Open)

> Running unit tests on Ruby 2.2
> --
>
> Key: AVRO-1775
> URL: https://issues.apache.org/jira/browse/AVRO-1775
> Project: Avro
>  Issue Type: Bug
>  Components: ruby
>Reporter: Martin Kleppmann
> Attachments: AVRO-1775-1.patch
>
>
> Ruby 2.2 [removed the test/unit framework from the standard 
> library|https://bugs.ruby-lang.org/issues/9711#note-12]. As the Avro Ruby 
> implementation uses it for its tests, we need to add a dependency on the 
> {{test-unit}} gem in order to run the tests in Ruby 2.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1775) Running unit tests on Ruby 2.2

[Martin Kleppmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Kleppmann updated AVRO-1775:
---
Attachment: AVRO-1775-1.patch

Attached patch that adds a dependency on the {{test-unit}} gem. This makes 
{{build.sh test}} work with Ruby 2.2, and seems to have no negative 
side-effects in other versions of Ruby (I tested 1.9.3 and 2.1.4).

> Running unit tests on Ruby 2.2
> --
>
> Key: AVRO-1775
> URL: https://issues.apache.org/jira/browse/AVRO-1775
> Project: Avro
>  Issue Type: Bug
>  Components: ruby
>Reporter: Martin Kleppmann
> Attachments: AVRO-1775-1.patch
>
>
> Ruby 2.2 [removed the test/unit framework from the standard 
> library|https://bugs.ruby-lang.org/issues/9711#note-12]. As the Avro Ruby 
> implementation uses it for its tests, we need to add a dependency on the 
> {{test-unit}} gem in order to run the tests in Ruby 2.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1572) Avro-C misses EOF on 4K boundary

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075115#comment-15075115
 ] 

ASF subversion and git services commented on AVRO-1572:
---

Commit 1722359 from [~martinkl] in branch 'avro/trunk'
[ https://svn.apache.org/r1722359 ]

AVRO-1572: C: Fix EOF handling on data files that are multiples of 4096 bytes. 
Contributed by Ben Walsh.

> Avro-C misses EOF on 4K boundary
> 
>
> Key: AVRO-1572
> URL: https://issues.apache.org/jira/browse/AVRO-1572
> Project: Avro
>  Issue Type: Bug
>  Components: c
>Reporter: Ben Walsh
> Attachments: AVRO-1572.patch
>
>
> As noted in AVRO-1405, Avro C is still missing EOF in the case that the sync 
> section ends exactly on a 4096 byte boundary (4096 bytes is the buffer size 
> used by the reader).
> I have fixed this bug (the suggested fix for AVRO-1364 actually works fine). 
> Also I added some more extensive testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)