[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-04 Thread Yibing Shi (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463713#comment-15463713
 ] 

Yibing Shi commented on AVRO-1891:
--

[~rdblue], you idea looks good to me!

{quote}
Maintain a thread-local reference to the current specific record class in 
GenericDatumReader
{quote}
I am also thinking about using a thread local variable, because my patch makes 
data reader/writer non thread-safe. We may need to keep a stack of class 
references, which will be helpful when there are nested records.

{quote}
# Add conversion lookup methods to GenericDatumReader that delegate to 
GenericData
# Override the conversion lookup methods in SpecificDatumReader that use the 
current record class's set of conversions instead.
{quote}
This is a great idea! Looking forward to your patch!
(For backward compatibility, the conversion lookup methods in 
SpecificDataumReader/Writer may need to fallback to those in 
GenericDatumReader/Writer if the generated class doesn't have the static 
conversion map -- for legacy classes I mean)

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-04 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463687#comment-15463687
 ] 

Ryan Blue commented on AVRO-1891:
-

I think all this requires is keeping a set of conversions that should be 
applied when reading or writing a specific class. Unlike generic where the 
conversions are determined by the data model at runtime, the conversions that 
should be used for a specific class are determined at compile time. We have the 
benefit of knowing that the compiler either added conversions for all instances 
of a logical type, or for none of them. So we only need to know the set of 
conversions the compiler had set up when a class was compiled.

Rather than relying on the set of conversions the SpecificData instance has 
configured, I think we should keep the set of conversions for the class being 
written or read. So we don't need to change how SpecificData looks up 
conversions, just the way the SpecificDatumReader/Writer does to avoid looking 
them up in the data model. (I agree with Doug and don't see an advantage of 
adding a conversion resolver.)

What about this:

* Maintain a thread-local reference to the current specific record class in 
GenericDatumReader
* Add a static conversion map to each specific class with its conversions 
(generated code)
* Add conversion lookup methods to GenericDatumReader that delegate to 
GenericData
* Override the conversion lookup methods in SpecificDatumReader that use the 
current record class's set of conversions instead.

This way, there are no changes to how the data model lookups work, little 
generated code (just an annotation to conversion map), and few changes to the 
datum reader and writers. What do you guys think? I think this would be a bit 
smaller patch. I'll try to put it together tomorrow if I have time.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
>

[jira] [Updated] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-09-04 Thread Ryan Blue (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1883:

   Resolution: Fixed
Fix Version/s: 1.8.2
   1.9.0
   Status: Resolved  (was: Patch Available)

I committed this patch. Thanks for fixing this, [~Yibing]!

(I tested this with AVRO-1908 also applied because it hasn't been merged yet.)

> Schema validator cannot find broken backwards compatibility in Union type 
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Critical
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   },
>   {
> "name": "address",
> "type": "string"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the 
> {{address}} field of {{rec1}} in the second one is not nullable. However, if 
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new 
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-09-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463606#comment-15463606
 ] 

ASF subversion and git services commented on AVRO-1883:
---

Commit 5acae5e10ee5d9d7d98b208e29a930f1e66f40ae in avro's branch 
refs/heads/master from [~rdblue]
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=5acae5e ]

AVRO-1883: Java: Fix incompatible schema detection nested in unions. 
Contributed by Yibing Shi.


> Schema validator cannot find broken backwards compatibility in Union type 
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   },
>   {
> "name": "address",
> "type": "string"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the 
> {{address}} field of {{rec1}} in the second one is not nullable. However, if 
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new 
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1908) IPC TestSpecificCompiler build is broken

2016-09-04 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463575#comment-15463575
 ] 

Ryan Blue commented on AVRO-1908:
-

[~busbey], could you review? Thanks!

> IPC TestSpecificCompiler build is broken
> 
>
> Key: AVRO-1908
> URL: https://issues.apache.org/jira/browse/AVRO-1908
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Ryan Blue
>
> AVRO-1884 changed {{SpecificCompiler.makePath}} to {{private 
> SpecificCompiler#makePath}}, which broke building the tests in the IPC 
> package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1908) IPC TestSpecificCompiler build is broken

2016-09-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463574#comment-15463574
 ] 

ASF GitHub Bot commented on AVRO-1908:
--

GitHub user rdblue opened a pull request:

https://github.com/apache/avro/pull/117

AVRO-1908: Fix TestSpecificCompiler reference to private method.

AVRO-1884 changed makePath to a private method from a package-private
static method. This broke the test that references the method in IPC.
The fix is to make the instance method package-private and update the
test to use an instance of SpecificCompiler.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rdblue/avro AVRO-1908-fix-build-after-1884

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit 42cc964207170ab71ab306429ecea642e54048a0
Author: Ryan Blue 
Date:   2016-09-04T21:54:40Z

AVRO-1908: Fix TestSpecificCompiler reference to private method.

AVRO-1884 changed makePath to a private method from a package-private
static method. This broke the test that references the method in IPC.
The fix is to make the instance method package-private and update the
test to use an instance of SpecificCompiler.




> IPC TestSpecificCompiler build is broken
> 
>
> Key: AVRO-1908
> URL: https://issues.apache.org/jira/browse/AVRO-1908
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Ryan Blue
>
> AVRO-1884 changed {{SpecificCompiler.makePath}} to {{private 
> SpecificCompiler#makePath}}, which broke building the tests in the IPC 
> package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] avro pull request #117: AVRO-1908: Fix TestSpecificCompiler reference to pri...

2016-09-04 Thread rdblue 
GitHub user rdblue opened a pull request:

https://github.com/apache/avro/pull/117

AVRO-1908: Fix TestSpecificCompiler reference to private method.

AVRO-1884 changed makePath to a private method from a package-private
static method. This broke the test that references the method in IPC.
The fix is to make the instance method package-private and update the
test to use an instance of SpecificCompiler.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rdblue/avro AVRO-1908-fix-build-after-1884

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit 42cc964207170ab71ab306429ecea642e54048a0
Author: Ryan Blue 
Date:   2016-09-04T21:54:40Z

AVRO-1908: Fix TestSpecificCompiler reference to private method.

AVRO-1884 changed makePath to a private method from a package-private
static method. This broke the test that references the method in IPC.
The fix is to make the instance method package-private and update the
test to use an instance of SpecificCompiler.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (AVRO-1908) IPC TestSpecificCompiler build is broken

2016-09-04 Thread Ryan Blue (JIRA) 
Ryan Blue created AVRO-1908:
---

 Summary: IPC TestSpecificCompiler build is broken
 Key: AVRO-1908
 URL: https://issues.apache.org/jira/browse/AVRO-1908
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Ryan Blue


AVRO-1884 changed {{SpecificCompiler.makePath}} to {{private 
SpecificCompiler#makePath}}, which broke building the tests in the IPC package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1874) py3 avro module import upsets logging level in host application

2016-09-04 Thread Ryan Blue (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1874:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for fixing this, [~torgebo]! I've merged your PR into master and it 
should be in the 1.8.2 release.

> py3 avro module import upsets logging level in host application
> ---
>
> Key: AVRO-1874
> URL: https://issues.apache.org/jira/browse/AVRO-1874
> Project: Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.8.1
> Environment: Mac OSX El Capitan, Macbook Pro,
> Anaconda Python v. 3.5.1
> Avro installed from source of Avro1.8.1/lang/py3
> (apache package "avro-src-1.8.1.tar.gz")
> using "sudo python setup.py install"
>Reporter: Torgeir Børresen
>Assignee: Torgeir Børresen
>Priority: Critical
> Fix For: 1.8.2
>
>
> When importing "avro.datafile" the logging level of the host application gets 
> overriden.
> In the simple example provided here: 
> https://github.com/torgebo/avro-1.8.1-logging-break
> the logging level is wrongfully set to "logging.WARNING" during execution 
> instead of "logging.INFO".
> The issue seems to be resolved by using module level loggers in the pattern of
> logger = logging.getLogger(__name__)
> and replacing current calls to the logger named "logging" as this logger 
> "logger" instead. This approach is described here: 
> https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial
> When setting logger across all avro source files, it is observed that the 
> application sets the logging level faithfully.
> 
> This issue was not observed with python version 2, although the recommended 
> way to resolve module level logging as described in the logging python docs 
> seems to be the same (ie. using the logging.getLogger method to access the 
> logger handle).
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1874) py3 avro module import upsets logging level in host application

2016-09-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463545#comment-15463545
 ] 

ASF subversion and git services commented on AVRO-1874:
---

Commit 64f898c4322ec641379c9916007303acfe817f84 in avro's branch 
refs/heads/master from Torgeir Boerresen
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=64f898c ]

AVRO-1874: Py3: Use recommended module-level logging.

The previous use of the logging library, using logging as logger
handler, unsets logging level set by host application.

Closes #105.


> py3 avro module import upsets logging level in host application
> ---
>
> Key: AVRO-1874
> URL: https://issues.apache.org/jira/browse/AVRO-1874
> Project: Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.8.1
> Environment: Mac OSX El Capitan, Macbook Pro,
> Anaconda Python v. 3.5.1
> Avro installed from source of Avro1.8.1/lang/py3
> (apache package "avro-src-1.8.1.tar.gz")
> using "sudo python setup.py install"
>Reporter: Torgeir Børresen
>Assignee: Torgeir Børresen
>Priority: Critical
> Fix For: 1.8.2
>
>
> When importing "avro.datafile" the logging level of the host application gets 
> overriden.
> In the simple example provided here: 
> https://github.com/torgebo/avro-1.8.1-logging-break
> the logging level is wrongfully set to "logging.WARNING" during execution 
> instead of "logging.INFO".
> The issue seems to be resolved by using module level loggers in the pattern of
> logger = logging.getLogger(__name__)
> and replacing current calls to the logger named "logging" as this logger 
> "logger" instead. This approach is described here: 
> https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial
> When setting logger across all avro source files, it is observed that the 
> application sets the logging level faithfully.
> 
> This issue was not observed with python version 2, although the recommended 
> way to resolve module level logging as described in the logging python docs 
> seems to be the same (ie. using the logging.getLogger method to access the 
> logger handle).
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1874) py3 avro module import upsets logging level in host application

2016-09-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463547#comment-15463547
 ] 

ASF GitHub Bot commented on AVRO-1874:
--

Github user asfgit closed the pull request at:

https://github.com/apache/avro/pull/105


> py3 avro module import upsets logging level in host application
> ---
>
> Key: AVRO-1874
> URL: https://issues.apache.org/jira/browse/AVRO-1874
> Project: Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.8.1
> Environment: Mac OSX El Capitan, Macbook Pro,
> Anaconda Python v. 3.5.1
> Avro installed from source of Avro1.8.1/lang/py3
> (apache package "avro-src-1.8.1.tar.gz")
> using "sudo python setup.py install"
>Reporter: Torgeir Børresen
>Assignee: Torgeir Børresen
>Priority: Critical
> Fix For: 1.8.2
>
>
> When importing "avro.datafile" the logging level of the host application gets 
> overriden.
> In the simple example provided here: 
> https://github.com/torgebo/avro-1.8.1-logging-break
> the logging level is wrongfully set to "logging.WARNING" during execution 
> instead of "logging.INFO".
> The issue seems to be resolved by using module level loggers in the pattern of
> logger = logging.getLogger(__name__)
> and replacing current calls to the logger named "logging" as this logger 
> "logger" instead. This approach is described here: 
> https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial
> When setting logger across all avro source files, it is observed that the 
> application sets the logging level faithfully.
> 
> This issue was not observed with python version 2, although the recommended 
> way to resolve module level logging as described in the logging python docs 
> seems to be the same (ie. using the logging.getLogger method to access the 
> logger handle).
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] avro pull request #105: AVRO-1874 Correct module level logging importation.

2016-09-04 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/avro/pull/105


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (AVRO-1884) Add set source file suffix function for generate non-java file #90

2016-09-04 Thread Ryan Blue (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1884:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Merged #90. Thanks for contributing, [~shijinkui]!

> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
> Fix For: 1.9.0, 1.8.2
>
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1884) Add set source file suffix function for generate non-java file #90

2016-09-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463506#comment-15463506
 ] 

ASF GitHub Bot commented on AVRO-1884:
--

Github user asfgit closed the pull request at:

https://github.com/apache/avro/pull/90


> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
> Fix For: 1.9.0, 1.8.2
>
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1884) Add set source file suffix function for generate non-java file #90

2016-09-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463505#comment-15463505
 ] 

ASF subversion and git services commented on AVRO-1884:
---

Commit e6d2c4d86490e6e24039365476c8eeb2ea1e in avro's branch 
refs/heads/master from [~shijinkui]
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=e6d2c4d ]

AVRO-1884: Java: Add method to set the compiler output suffix.

Use when generating non-Java files with custom templates. For example:

```
compiler.setSuffix(".scala")
compiler.setTemplateDir(scalaTemplatePath)
compiler.compileToDestination(file, new File("src/main/scala/"))
```

Closes #90.


> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
> Fix For: 1.9.0, 1.8.2
>
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] avro pull request #90: AVRO-1884: Add set source file suffix function for ge...

2016-09-04 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/avro/pull/90


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (AVRO-1879) Make conversions field transient in compiled SpecificRecord

2016-09-04 Thread Ryan Blue (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1879:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Merged #108. Thanks [~mwong]!

> Make conversions field transient in compiled SpecificRecord
> ---
>
> Key: AVRO-1879
> URL: https://issues.apache.org/jira/browse/AVRO-1879
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Michael Wong
>Assignee: Michael Wong
> Fix For: 1.9.0, 1.8.2
>
> Attachments: conversions-transient.patch
>
>
> Add a transient modifier to the conversions field as such
> {code}
> private final transient org.apache.avro.Conversion[] conversions
> {code}
> This will allow Serializers and type inference systems like Flink's 
> TypeExtractor to not consider the {{conversions}} field and not try to 
> serialize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1879) Make conversions field transient in compiled SpecificRecord

2016-09-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463494#comment-15463494
 ] 

ASF subversion and git services commented on AVRO-1879:
---

Commit c37d05e87957d7b46c32c5fadcb569dc9be44117 in avro's branch 
refs/heads/master from [~mwong]
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=c37d05e ]

AVRO-1879: Make conversions field static.


> Make conversions field transient in compiled SpecificRecord
> ---
>
> Key: AVRO-1879
> URL: https://issues.apache.org/jira/browse/AVRO-1879
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Michael Wong
>Assignee: Michael Wong
> Fix For: 1.9.0, 1.8.2
>
> Attachments: conversions-transient.patch
>
>
> Add a transient modifier to the conversions field as such
> {code}
> private final transient org.apache.avro.Conversion[] conversions
> {code}
> This will allow Serializers and type inference systems like Flink's 
> TypeExtractor to not consider the {{conversions}} field and not try to 
> serialize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

2016-09-04 Thread Ryan Blue (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1704:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

I committed the last patch with the spec changes, which closes out this issue. 
Thanks [~nielsbasjes], [~cutting], [~busbey], and [~dasch] for making this 
happen!

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1704) Standardized format for encoding messages with Avro

2016-09-04 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463478#comment-15463478
 ] 

Ryan Blue commented on AVRO-1704:
-

Thanks for reviewing!

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

2016-09-04 Thread Sean Busbey (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463382#comment-15463382
 ] 

Sean Busbey commented on AVRO-607:
--

I marked the helpwanted task as done. Thanks!

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
>  Labels: newbie, patch
> Fix For: 1.8.2
>
> Attachments: 
> 0001-AVRO-607-Changed-SpecificData.getSchema-to-use-a-thr.patch, 
> AVRO-607.patch, AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1704) Standardized format for encoding messages with Avro

2016-09-04 Thread Sean Busbey (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463379#comment-15463379
 ] 

Sean Busbey commented on AVRO-1704:
---

+1 on AVRO-1704.4.patch

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1719) Avro fails to build against Boost 1.59.0

2016-09-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462601#comment-15462601
 ] 

ASF GitHub Bot commented on AVRO-1719:
--

Github user Romain-Geissler-1A closed the pull request at:

https://github.com/apache/avro/pull/51


> Avro fails to build against Boost 1.59.0
> 
>
> Key: AVRO-1719
> URL: https://issues.apache.org/jira/browse/AVRO-1719
> Project: Avro
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.7.7
>Reporter: Tim Smith
>Assignee: Romain Geissler
>Priority: Blocker
> Fix For: 1.7.8, 1.8.2
>
>
> Avro fails to build on OS X with Boost 1.59.0, dying on errors about 
> undeclared BOOST_ identifiers.
> Build logs are here: 
> https://gist.github.com/anonymous/03736608223d42f45ab1#file-02-make-L180
> Homebrew is tracking packages which fail to build against Boost 1.59.0 at 
> https://github.com/Homebrew/homebrew/pull/42960.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] avro pull request #51: [AVRO-1719] Fix test when using Boost >= 1.59.

2016-09-04 Thread Romain-Geissler-1A 
Github user Romain-Geissler-1A closed the pull request at:

https://github.com/apache/avro/pull/51


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---