date:20190820

Re: [VOTE] Release Apache Avro 1.9.1 RC1

2019-08-20 Thread Brian Lachniet

It sounds like an rc2 is imminent, but I went ahead and ran some basic
checks on the C# bindings. I didn't find anything unusual. I published
Apache.Avro
v1.9.1-rc1  and
Apache.Avro.Tools
v1.9.1-rc1  to
nuget.org.

On Tue, Aug 20, 2019 at 3:44 PM Driesprong, Fokko 
wrote:

> Thanks, Dan for giving it a try. My mistake, this file comes from the
> test-suite. I'll clean up the repo and publish a new package.
>
> Cheers, Fokko
>
> Op di 20 aug. 2019 om 21:41 schreef Daniel Kulp :
>
> > Java doesn’t build.There is an empty file:
> >
> >
> >
> lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc
> >
> > In the tar that doesn’t seem to be in the git repo.  It’s causing a
> > problem with RAT since there isn’t a license header or anything.   I’m
> not
> > sure where that file came from.
> >
> >
> > Dan
> >
> >
> >
> >
> > > On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko 
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm delighted to propose the following RC to be released as official
> > Apache
> > > Avro 1.9.1 release.
> > >
> > > The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> > > * This corresponds to the tag: release-1.9.1-rc1
> > > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> > >
> > > The release tarball, signature, and checksums are here:
> > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > >
> > > Binary artifacts for Java are staged in Nexus here:
> > > *
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> > >
> > > This release includes 31 Jira issues:
> > >
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > > * Most important, fix regression issues:
> > >  * Java: decoding schema's:
> > https://jira.apache.org/jira/browse/AVRO-2400
> > >  * .Net: Performance issue:
> > https://jira.apache.org/jira/browse/AVRO-2396
> > > * Java: Make org.apache.avro.Schema serializable
> > > * Java: Ability to add custom object to Velocity templating
> > > * Improved interoperability testing
> > > * Removed NPE's
> > > * Upgrade dependencies to latest to the latest version
> > > * And many more :-)
> > >
> > > Please download, verify, and test. This vote will remain open for at
> > least
> > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > midnight
> > > on Saturday, 24 August 2019.
> > >
> > > [ ] +1 Release this as Apache Avro 1.9.1
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Consider this a +1 (binding) from my side:
> > > * Compiled against Divolte collector and Iceberg
> > >
> > > Cheers, Fokko Driesprong
> >
> > --
> > Daniel Kulp
> > dk...@apache.org  - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com 
> >
>


-- 

[image: 51b630b05e01a6d5134ccfd520f547c4.png]

Brian Lachniet

Software Engineer

E: blachn...@gmail.com | blachniet.com

Re: [VOTE] Release Apache Avro 1.9.1 RC1

2019-08-20 Thread Driesprong, Fokko

Thanks, Dan for giving it a try. My mistake, this file comes from the
test-suite. I'll clean up the repo and publish a new package.

Cheers, Fokko

Op di 20 aug. 2019 om 21:41 schreef Daniel Kulp :

> Java doesn’t build.There is an empty file:
>
>
> lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc
>
> In the tar that doesn’t seem to be in the git repo.  It’s causing a
> problem with RAT since there isn’t a license header or anything.   I’m not
> sure where that file came from.
>
>
> Dan
>
>
>
>
> > On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko 
> wrote:
> >
> > Hi everyone,
> >
> > I'm delighted to propose the following RC to be released as official
> Apache
> > Avro 1.9.1 release.
> >
> > The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> > * This corresponds to the tag: release-1.9.1-rc1
> > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> >
> > This release includes 31 Jira issues:
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > * Most important, fix regression issues:
> >  * Java: decoding schema's:
> https://jira.apache.org/jira/browse/AVRO-2400
> >  * .Net: Performance issue:
> https://jira.apache.org/jira/browse/AVRO-2396
> > * Java: Make org.apache.avro.Schema serializable
> > * Java: Ability to add custom object to Velocity templating
> > * Improved interoperability testing
> > * Removed NPE's
> > * Upgrade dependencies to latest to the latest version
> > * And many more :-)
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close it on or about
> > midnight
> > on Saturday, 24 August 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.1
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (binding) from my side:
> > * Compiled against Divolte collector and Iceberg
> >
> > Cheers, Fokko Driesprong
>
> --
> Daniel Kulp
> dk...@apache.org  - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com 
>

Re: [VOTE] Release Apache Avro 1.9.1 RC1

2019-08-20 Thread Daniel Kulp

Java doesn’t build.There is an empty file:

lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc

In the tar that doesn’t seem to be in the git repo.  It’s causing a problem 
with RAT since there isn’t a license header or anything.   I’m not sure where 
that file came from.


Dan




> On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko  wrote:
> 
> Hi everyone,
> 
> I'm delighted to propose the following RC to be released as official Apache
> Avro 1.9.1 release.
> 
> The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> * This corresponds to the tag: release-1.9.1-rc1
> * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> 
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> 
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/avro/KEYS
> 
> Binary artifacts for Java are staged in Nexus here:
> *
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> 
> This release includes 31 Jira issues:
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> * Most important, fix regression issues:
>  * Java: decoding schema's: https://jira.apache.org/jira/browse/AVRO-2400
>  * .Net: Performance issue: https://jira.apache.org/jira/browse/AVRO-2396
> * Java: Make org.apache.avro.Schema serializable
> * Java: Ability to add custom object to Velocity templating
> * Improved interoperability testing
> * Removed NPE's
> * Upgrade dependencies to latest to the latest version
> * And many more :-)
> 
> Please download, verify, and test. This vote will remain open for at least
> 72 hours. Given sufficient votes, I would like to close it on or about
> midnight
> on Saturday, 24 August 2019.
> 
> [ ] +1 Release this as Apache Avro 1.9.1
> [ ] +0
> [ ] -1 Do not release this because...
> 
> Consider this a +1 (binding) from my side:
> * Compiled against Divolte collector and Iceberg
> 
> Cheers, Fokko Driesprong

-- 
Daniel Kulp
dk...@apache.org  - http://dankulp.com/blog 

Talend Community Coder - http://talend.com

[VOTE] Release Apache Avro 1.9.1 RC1

2019-08-20 Thread Driesprong, Fokko

Hi everyone,

I'm delighted to propose the following RC to be released as official Apache
Avro 1.9.1 release.

The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
* This corresponds to the tag: release-1.9.1-rc1
* https://github.com/apache/avro/releases/tag/release-1.9.1-rc1

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/

This release includes 31 Jira issues:
https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
* Most important, fix regression issues:
  * Java: decoding schema's: https://jira.apache.org/jira/browse/AVRO-2400
  * .Net: Performance issue: https://jira.apache.org/jira/browse/AVRO-2396
* Java: Make org.apache.avro.Schema serializable
* Java: Ability to add custom object to Velocity templating
* Improved interoperability testing
* Removed NPE's
* Upgrade dependencies to latest to the latest version
* And many more :-)

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 24 August 2019.

[ ] +1 Release this as Apache Avro 1.9.1
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (binding) from my side:
* Compiled against Divolte collector and Iceberg

Cheers, Fokko Driesprong

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AVRO-2514:
---
Fix Version/s: (was: 1.9.1)
   1.10.0

> Making Avro serialization error message more actionable with schema name and 
> field name
> ---
>
> Key: AVRO-2514
> URL: https://issues.apache.org/jira/browse/AVRO-2514
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.6.3, 1.9.0, 1.8.2
>Reporter: Jagannathrao Mudda
>Priority: Major
> Fix For: 1.10.0
>
>
> In certain scenarios when there is a serialization error, avro serializer 
> gives following error messages that are not actionable without any schema 
> name and/field name in it.
>  * java.lang.Double cannot be cast to java.lang.Long
>  * Not an enum: null
>  * ClassCastException 
>  * ..
> If there is a filed name of specific schema that is causing these 
> serialization errors, that helps to fix such issues quickly.
>  
> *Proposed Changes:*
> The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
> updated to handle "ClassCastException" along with "NullPointerException" to 
> throw an exception with schema name and field name.
> The "writeEnum", "writeField", and "error" method can be enhanced to add 
> field names so that error message can have the field that is causing the 
> issue.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (AVRO-1852) Make org.apache.avro.Schema serializable (java.io.Serializable)

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong reassigned AVRO-1852:
--

Assignee: Zoltan Farkas

> Make org.apache.avro.Schema serializable (java.io.Serializable)
> ---
>
> Key: AVRO-1852
> URL: https://issues.apache.org/jira/browse/AVRO-1852
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
>Priority: Minor
> Fix For: 1.10.0, 1.9.1
>
>
> here is a commit describing the implementation: 
> https://github.com/zolyfarkas/avro/commit/867f4d6a0f2e65a4ca8084f02b0d704a3acdb9d0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (AVRO-2513) Install Perl packages over https

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2513.

Resolution: Fixed

> Install Perl packages over https
> 
>
> Key: AVRO-2513
> URL: https://issues.apache.org/jira/browse/AVRO-2513
> Project: Apache Avro
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

2019-08-20 Thread Jagannathrao Mudda (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jagannathrao Mudda updated AVRO-2514:
-
Description: 
In certain scenarios when there is a serialization error, avro serializer gives 
following error messages that are not actionable without any schema name 
and/field name in it.
 * java.lang.Double cannot be cast to java.lang.Long
 * Not an enum: null
 * ClassCastException 
 * ..

If there is a filed name of specific schema that is causing these serialization 
errors, that helps to fix such issues quickly.

 

*Proposed Changes:*

The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
updated to handle "ClassCastException" along with "NullPointerException" to 
throw an exception with schema name and field name.

The "writeEnum", "writeField", and "error" method can be enhanced to add field 
names so that error message can have the field that is causing the issue.

 

 

  was:
In certain scenarios when there is a serialization error, avro serializer gives 
following error messages that are not actionable without any schema name 
and/field name in it.
 * java.lang.Double cannot be cast to java.lang.Long
 * Not an enum: null
 * ClassCastException 
 * ..

If there is a filed name of specific schema that is causing these serialization 
errors, that helps to fix such issues quickly.

 

*Proposed Changes:*

The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
updated to handle "ClassCastException" along with "NullPointerException" to 
throw an exception with schema name and field name.

The "writeEnum" and "error" method can be enhanced to add field names so that 
error message can have the field that is causing the issue.

 

 


> Making Avro serialization error message more actionable with schema name and 
> field name
> ---
>
> Key: AVRO-2514
> URL: https://issues.apache.org/jira/browse/AVRO-2514
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.6.3, 1.9.0, 1.8.2
>Reporter: Jagannathrao Mudda
>Priority: Major
> Fix For: 1.9.1
>
>
> In certain scenarios when there is a serialization error, avro serializer 
> gives following error messages that are not actionable without any schema 
> name and/field name in it.
>  * java.lang.Double cannot be cast to java.lang.Long
>  * Not an enum: null
>  * ClassCastException 
>  * ..
> If there is a filed name of specific schema that is causing these 
> serialization errors, that helps to fix such issues quickly.
>  
> *Proposed Changes:*
> The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
> updated to handle "ClassCastException" along with "NullPointerException" to 
> throw an exception with schema name and field name.
> The "writeEnum", "writeField", and "error" method can be enhanced to add 
> field names so that error message can have the field that is causing the 
> issue.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

2019-08-20 Thread Jagannathrao Mudda (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jagannathrao Mudda updated AVRO-2514:
-
Description: 
In certain scenarios when there is a serialization error, avro serializer gives 
following error messages that are not actionable without any schema name 
and/field name in it.
 * java.lang.Double cannot be cast to java.lang.Long
 * Not an enum: null
 * ClassCastException 
 * ..

If there is a filed name of specific schema that is causing these serialization 
errors, that helps to fix such issues quickly.

 

*Proposed Changes:*

The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
updated to handle "ClassCastException" along with "NullPointerException" to 
throw an exception with schema name and field name.

The "writeEnum" and "error" method can be enhanced to add field names so that 
error message can have the field that is causing the issue.

 

 

  was:
In certain scenarios when there is a serialization error, avro serializer gives 
following error messages that are not actionable without any schema name 
and/field name in it.
 * java.lang.Double cannot be cast to java.lang.Long
 * Not an enum: null
 * ClassCastException 
 * ..

If there is a filed name of specific schema that is causing these serialization 
errors, that helps to fix such issues quickly.

 

 

 


> Making Avro serialization error message more actionable with schema name and 
> field name
> ---
>
> Key: AVRO-2514
> URL: https://issues.apache.org/jira/browse/AVRO-2514
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.6.3, 1.9.0, 1.8.2
>Reporter: Jagannathrao Mudda
>Priority: Major
> Fix For: 1.9.1
>
>
> In certain scenarios when there is a serialization error, avro serializer 
> gives following error messages that are not actionable without any schema 
> name and/field name in it.
>  * java.lang.Double cannot be cast to java.lang.Long
>  * Not an enum: null
>  * ClassCastException 
>  * ..
> If there is a filed name of specific schema that is causing these 
> serialization errors, that helps to fix such issues quickly.
>  
> *Proposed Changes:*
> The "writeWithoutConversion" method of "GenericDatumWriter"  class can be 
> updated to handle "ClassCastException" along with "NullPointerException" to 
> throw an exception with schema name and field name.
> The "writeEnum" and "error" method can be enhanced to add field names so that 
> error message can have the field that is causing the issue.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

2019-08-20 Thread Jagannathrao Mudda (Jira)

Jagannathrao Mudda created AVRO-2514:


 Summary: Making Avro serialization error message more actionable 
with schema name and field name
 Key: AVRO-2514
 URL: https://issues.apache.org/jira/browse/AVRO-2514
 Project: Apache Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.2, 1.9.0, 1.6.3
Reporter: Jagannathrao Mudda
 Fix For: 1.9.1


In certain scenarios when there is a serialization error, avro serializer gives 
following error messages that are not actionable without any schema name 
and/field name in it.
 * java.lang.Double cannot be cast to java.lang.Long
 * Not an enum: null
 * ClassCastException 
 * ..

If there is a filed name of specific schema that is causing these serialization 
errors, that helps to fix such issues quickly.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2512) writerField is always null

2019-08-20 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911441#comment-16911441
 ] 

Hudson commented on AVRO-2512:
--

SUCCESS: Integrated in Jenkins build AvroJava #726 (See 
[https://builds.apache.org/job/AvroJava/726/])
AVRO-2512: writerField is always null (fokko: 
[https://github.com/apache/avro/commit/058cfefadeaa642d1ce17727ded149b3a01b6de5])
* (edit) lang/java/avro/src/main/java/org/apache/avro/SchemaCompatibility.java
AVRO-2512: Fix formatting (fokko: 
[https://github.com/apache/avro/commit/eda435931b8dbd913b98567814494ddfe4bea016])
* (edit) lang/java/avro/src/main/java/org/apache/avro/SchemaCompatibility.java


> writerField is always null
> --
>
> Key: AVRO-2512
> URL: https://issues.apache.org/jira/browse/AVRO-2512
> Project: Apache Avro
>  Issue Type: Task
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2513) Install Perl packages over https

2019-08-20 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911442#comment-16911442
 ] 

Hudson commented on AVRO-2513:
--

SUCCESS: Integrated in Jenkins build AvroJava #726 (See 
[https://builds.apache.org/job/AvroJava/726/])
AVRO-2513: Install Perl packages over https (fokko: 
[https://github.com/apache/avro/commit/d1eac15926e73d1bdc91867881b13b93d7fdf3e0])
* (edit) share/docker/Dockerfile


> Install Perl packages over https
> 
>
> Key: AVRO-2513
> URL: https://issues.apache.org/jira/browse/AVRO-2513
> Project: Apache Avro
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2512) writerField is always null

2019-08-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911400#comment-16911400
 ] 

ASF subversion and git services commented on AVRO-2512:
---

Commit eda435931b8dbd913b98567814494ddfe4bea016 in avro's branch 
refs/heads/master from Fokko Driesprong
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=eda4359 ]

AVRO-2512: Fix formatting


> writerField is always null
> --
>
> Key: AVRO-2512
> URL: https://issues.apache.org/jira/browse/AVRO-2512
> Project: Apache Avro
>  Issue Type: Task
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2512) writerField is always null

2019-08-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911362#comment-16911362
 ] 

ASF subversion and git services commented on AVRO-2512:
---

Commit 058cfefadeaa642d1ce17727ded149b3a01b6de5 in avro's branch 
refs/heads/master from Fokko Driesprong
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=058cfef ]

AVRO-2512: writerField is always null


> writerField is always null
> --
>
> Key: AVRO-2512
> URL: https://issues.apache.org/jira/browse/AVRO-2512
> Project: Apache Avro
>  Issue Type: Task
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2513) Install Perl packages over https

2019-08-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911363#comment-16911363
 ] 

ASF subversion and git services commented on AVRO-2513:
---

Commit d1eac15926e73d1bdc91867881b13b93d7fdf3e0 in avro's branch 
refs/heads/master from Fokko Driesprong
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=d1eac15 ]

AVRO-2513: Install Perl packages over https


> Install Perl packages over https
> 
>
> Key: AVRO-2513
> URL: https://issues.apache.org/jira/browse/AVRO-2513
> Project: Apache Avro
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2513) Install Perl packages over https

2019-08-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911354#comment-16911354
 ] 

ASF subversion and git services commented on AVRO-2513:
---

Commit b8b82de876b3e38519486952c56c74113156dc0e in avro's branch 
refs/heads/branch-1.9 from Fokko Driesprong
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=b8b82de ]

AVRO-2513: Install Perl packages over https

(cherry picked from commit d1eac15926e73d1bdc91867881b13b93d7fdf3e0)


> Install Perl packages over https
> 
>
> Key: AVRO-2513
> URL: https://issues.apache.org/jira/browse/AVRO-2513
> Project: Apache Avro
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AVRO-2513) Install Perl packages over https

2019-08-20 Thread Fokko Driesprong (Jira)

Fokko Driesprong created AVRO-2513:
--

 Summary: Install Perl packages over https
 Key: AVRO-2513
 URL: https://issues.apache.org/jira/browse/AVRO-2513
 Project: Apache Avro
  Issue Type: Task
  Components: build
Affects Versions: 1.9.0
Reporter: Fokko Driesprong
Assignee: Fokko Driesprong
 Fix For: 1.10.0, 1.9.1






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (AVRO-2491) Schema.createRecord allows to create non parseable Schemas

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2491.

Resolution: Fixed

> Schema.createRecord allows to create non parseable Schemas
> --
>
> Key: AVRO-2491
> URL: https://issues.apache.org/jira/browse/AVRO-2491
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> Avro's API allows to create Schema objects that are not parseable by Avro. We 
> probably must document this behavior in detail and deprecate the method that 
> allows to do this for a future release. Example of creating an unparseable 
> schema (it does not have a name and no public API to assign it a name):
> {quote}Schema schema = Schema.createRecord(new ArrayList<>());
>  Schema parsedSchema = new Schema.Parser().parse(schema.toString());
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AVRO-2486) fwrite error ignored

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AVRO-2486:
---
Fix Version/s: (was: 1.9.1)
   1.10.0

> fwrite error ignored
> 
>
> Key: AVRO-2486
> URL: https://issues.apache.org/jira/browse/AVRO-2486
> Project: Apache Avro
>  Issue Type: Bug
>  Components: c
>Affects Versions: 1.8.0, 1.8.1, 1.9.0, 1.8.2
>Reporter: mathew boorman
>Priority: Critical
> Fix For: 1.10.0
>
>
> C library: Errors from fwrite are ignored, resulting in corrupt files.
>  
> BugFix pull request:  https://github.com/apache/avro/pull/594



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (AVRO-2414) Fix flaky TestDataFileReader::getNumberOfOpenFileDescriptors on JDK11

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2414.

Resolution: Cannot Reproduce

> Fix flaky TestDataFileReader::getNumberOfOpenFileDescriptors on JDK11
> -
>
> Key: AVRO-2414
> URL: https://issues.apache.org/jira/browse/AVRO-2414
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 1.9.1
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AVRO-2512) writerField is always null

2019-08-20 Thread Fokko Driesprong (Jira)

Fokko Driesprong created AVRO-2512:
--

 Summary: writerField is always null
 Key: AVRO-2512
 URL: https://issues.apache.org/jira/browse/AVRO-2512
 Project: Apache Avro
  Issue Type: Task
Affects Versions: 1.9.0
Reporter: Fokko Driesprong






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AVRO-2511) Avro Java DataFileWriter Flush() does not flush the buffer to disk

2019-08-20 Thread Asen Milchev Kolev (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Asen Milchev Kolev updated AVRO-2511:
-
Description: 
If you try to use flush() with output stream instead of a file, the buffer is 
not flushed to disk. Here is an example how I'm using it and there is no chance 
to see data flushed to disk! Is that by design or it is a bug? I really need 
this in order to determine file size and create a new one when max file size is 
reached!
{code:java}

try (DataFileWriter dataFileWriter = new DataFileWriter<>(new 
GenericDatumWriter<>());                     DataFileStream 
dataFileStreamReader = new DataFileStream<>(is, new 
GenericDatumReader()))               
 {                    
dataFileWriter.setFlushOnEveryBlock(true);                    
FSDataOutputStream hdfsOutputStream = null;                    
dataFileWriter.setCodec(codecFactory);                    
Schema schema = dataFileStreamReader.getSchema();                    
if (fileMode.equals(FileMode.APPEND))
{
  FileContext fc = FileContext.getFileContext(hdfsConfiguration);
  hdfsOutputStream = fileSystem.append(hdfsPath);
  dataFileWriter.appendTo(new AvroFSInput(fc, hdfsPath), hdfsOutputStream);
}
else
{
  hdfsOutputStream = fileSystem.create(hdfsPath);                        
  fileManager.setCreationTime(hdfsPath);
  dataFileWriter.create(schema, hdfsOutputStream);
}

GenericRecord genericRecord = null;
    while (dataFileStreamReader.hasNext())
{
  if (fileManager.isLimitsReached())
  {                            
IOUtils.closeStream(dataFileWriter);
fileSystem.rename(hdfsPath, fileManager.getFinalPath(hdfsPath));
        LOG.info("Avro write completed for {0}", hdfsPath.toString());
hdfsPath = fileManager.getPath();
storePaths.add(hdfsPath);
hdfsOutputStream = fileSystem.create(hdfsPath);
fileManager.setCreationTime(hdfsPath);
dataFileWriter.setCodec(codecFactory);                            
dataFileWriter.create(schema, hdfsOutputStream);
LOG.info("Initiate Avro write to {0}", hdfsPath.toString());
      }                        

  genericRecord = dataFileStreamReader.next(genericRecord);
      dataFileWriter.append(genericRecord);
  // doesn't work at all when we are using streams (i.e. 
dataFileWriter.create(schema, hdfsOutputStream);) and not directly files (i.e. 
dataFileWriter.create(schema, new File("..."));)! 
  dataFileWriter.flush();                        
  fileManager.updateEntryCount(hdfsPath);
  }
{code}

  was:
If you try to use flush() with output stream instead of a file, the buffer is 
not flushed to disk. Here is an example how I'm using it and there is no chance 
to see data flushed to disk! Is that by design or it is a bug? I really need 
this in order to determine file size and create a new one when max file size is 
reached!
{code:java}

try (DataFileWriter dataFileWriter = new DataFileWriter<>(new 
GenericDatumWriter<>());                     DataFileStream 
dataFileStreamReader = new DataFileStream<>(is, new 
GenericDatumReader()))                {                    
dataFileWriter.setFlushOnEveryBlock(true);                    
FSDataOutputStream hdfsOutputStream = null;                    
dataFileWriter.setCodec(codecFactory);                    Schema schema = 
dataFileStreamReader.getSchema();                    if 
(fileMode.equals(FileMode.APPEND))                    {                        
FileContext fc = FileContext.getFileContext(hdfsConfiguration);                 
       hdfsOutputStream = fileSystem.append(hdfsPath);                        
dataFileWriter.appendTo(new AvroFSInput(fc, hdfsPath), hdfsOutputStream);       
             }                    else                    {                     
   hdfsOutputStream = fileSystem.create(hdfsPath);                        
fileManager.setCreationTime(hdfsPath);                        
dataFileWriter.create(schema, hdfsOutputStream);                    }
                    GenericRecord genericRecord = null;                    
while (dataFileStreamReader.hasNext())                    {                     
   if (fileManager.isLimitsReached())                        {                  
          IOUtils.closeStream(dataFileWriter);                            
fileSystem.rename(hdfsPath, fileManager.getFinalPath(hdfsPath));                
            LOG.info("Avro write completed for {0}", hdfsPath.toString());      
                      hdfsPath = fileManager.getPath();                         
   storePaths.add(hdfsPath);                            hdfsOutputStream = 
fileSystem.create(hdfsPath);                            
fileManager.setCreationTime(hdfsPath);                            
dataFileWriter.setCodec(codecFactory);

[jira] [Commented] (AVRO-2468) Fix broken data interoperability on the Perl bindings

2019-08-20 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911171#comment-16911171
 ] 

Hudson commented on AVRO-2468:
--

SUCCESS: Integrated in Jenkins build AvroJava #725 (See 
[https://builds.apache.org/job/AvroJava/725/])
AVRO-2468: Fix broken data interoperability on the Perl bindings (#582) (fokko: 
[https://github.com/apache/avro/commit/02dbe25e35d5ffc644f049cf602f88c1b137ea29])
* (edit) lang/perl/build.sh
* (add) lang/perl/xt/interop.t
* (edit) build.sh
* (add) lang/perl/xt/schema.t
* (add) lang/perl/share/interop-data-generate
* (edit) lang/perl/lib/Avro/Schema.pm


> Fix broken data interoperability on the Perl bindings
> -
>
> Key: AVRO-2468
> URL: https://issues.apache.org/jira/browse/AVRO-2468
> Project: Apache Avro
>  Issue Type: Bug
>  Components: interop, perl
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Critical
> Fix For: 1.10.0
>
>
> I found some data interop problems on the Perl bindings.
> 1. They fail to parse a schema if there's an array/map/union which contains 
> named types with a simple (not fully-qualified) name in it. For example, they 
> can't parse {{share/test/schemas/interop.avsc}} or 
> {{share/schemas/org/apache/avro/data/Json.avsc}}, because they have a named 
> type called "Node" or "Json" respectively in arrays/maps. This seems because 
> the parser doesn't take namespace into consideration in parsing 
> array/map/union.
> {code}
> $ cd lang/perl
> $ perl -Ilib -de 1
> (snip)
>   DB<1> open FH, '../../share/test/schemas/interop.avsc'; local $/ = undef; 
> $s = ; close FH; print $s
> {"type": "record", "name":"Interop", "namespace": "org.apache.avro",
>   "fields": [
>   {"name": "intField", "type": "int"},
>   {"name": "longField", "type": "long"},
>   {"name": "stringField", "type": "string"},
>   {"name": "boolField", "type": "boolean"},
>   {"name": "floatField", "type": "float"},
>   {"name": "doubleField", "type": "double"},
>   {"name": "bytesField", "type": "bytes"},
>   {"name": "nullField", "type": "null"},
>   {"name": "arrayField", "type": {"type": "array", "items": "double"}},
>   {"name": "mapField", "type":
>{"type": "map", "values":
> {"type": "record", "name": "Foo",
>  "fields": [{"name": "label", "type": "string"}]}}},
>   {"name": "unionField", "type":
>["boolean", "double", {"type": "array", "items": "bytes"}]},
>   {"name": "enumField", "type":
>{"type": "enum", "name": "Kind", "symbols": ["A","B","C"]}},
>   {"name": "fixedField", "type":
>{"type": "fixed", "name": "MD5", "size": 16}},
>   {"name": "recordField", "type":
>{"type": "record", "name": "Node",
> "fields": [
> {"name": "label", "type": "string"},
> {"name": "children", "type": {"type": "array", "items": 
> "Node"}}]}}
>   ]
> }
>   DB<2> use Avro::Schema; Avro::Schema->parse($s)
> Not a primitive type Node at lib/Avro/Schema.pm line 257.
> {code}
> 2. They encode the size for a fixed type as a string rather than a number, so 
> other language bindings fail to parse it.
> {code}
> $ cd lang/perl 
> $ perl -Ilib -de 1
> (snip)
>   DB<1> use Avro::Schema; $s = Avro::Schema->parse('{"type": "fixed", "size": 
> 16, "name": "md5"}')
>   DB<2> open($fh, '>/tmp/output')
>   DB<3> use Avro::DataFileWriter; $w = Avro::DataFileWriter->new(fh => $fh, 
> writer_schema => $s)
>   DB<4> $w->print('0123456789abcdef')
>   DB<5> $w->close
> {code}
> {code}
> $ ipython
> (snip)
> In [1]: from avro.datafile import DataFileReader
> In [2]: from avro.io import DatumReader
> In [3]: DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> ---
> AvroException Traceback (most recent call last)
>  in ()
> > 1 DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> /home/sekikn/repo/avro/lang/py/src/avro/datafile.pyc in __init__(self, 
> reader, datum_reader)
> 255 # get ready to read
> 256 self._block_count = 0
> --> 257 self.datum_reader.writers_schema = 
> schema.parse(self.get_meta(SCHEMA_KEY))
> 258 
> 259   def __enter__(self):
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in parse(json_string)
> 984 
> 985   # construct the Avro Schema object
> --> 986   return make_avsc_object(json_data, names)
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in 
> make_avsc_object(json_data, names)
> 931   scale = 0 if json_data.get('scale') is None else 
> json_data.get('scale')
> 932   return FixedDecimalSchema(size, name, precision, scale, 
> namespace, names, other_props)
> --> 933 return

[jira] [Commented] (AVRO-2506) Fix warnings in creating the Ruby distribution

2019-08-20 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911172#comment-16911172
 ] 

Hudson commented on AVRO-2506:
--

SUCCESS: Integrated in Jenkins build AvroJava #725 (See 
[https://builds.apache.org/job/AvroJava/725/])
AVRO-2506: Fix warnings in creating the Ruby distribution (#607) (fokko: 
[https://github.com/apache/avro/commit/f17c77f20b74862b41bbc308619294d35871c518])
* (edit) lang/ruby/Rakefile


> Fix warnings in creating the Ruby distribution
> --
>
> Key: AVRO-2506
> URL: https://issues.apache.org/jira/browse/AVRO-2506
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: build
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Fix For: 1.10.0
>
>
> {code}
> sekikn@a80af4ec301b:~/avro/lang/ruby$ ./build.sh dist
> (snip)
> WARNING:  license value 'Apache License 2.0 (Apache-2.0)' is invalid.  Use a 
> license identifier from
> http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
> Did you mean 'Apache-2.0'?
> WARNING:  open-ended dependency on multi_json (>= 0) is not recommended
>   if multi_json is semantically versioned, use:
> add_runtime_dependency 'multi_json', '~> 0'
> WARNING:  See http://guides.rubygems.org/specification-reference/ for help
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AVRO-2506) Fix warnings in creating the Ruby distribution

2019-08-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AVRO-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911165#comment-16911165
 ] 

ASF subversion and git services commented on AVRO-2506:
---

Commit f17c77f20b74862b41bbc308619294d35871c518 in avro's branch 
refs/heads/master from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=f17c77f ]

AVRO-2506: Fix warnings in creating the Ruby distribution (#607)



> Fix warnings in creating the Ruby distribution
> --
>
> Key: AVRO-2506
> URL: https://issues.apache.org/jira/browse/AVRO-2506
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: build
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
>
> {code}
> sekikn@a80af4ec301b:~/avro/lang/ruby$ ./build.sh dist
> (snip)
> WARNING:  license value 'Apache License 2.0 (Apache-2.0)' is invalid.  Use a 
> license identifier from
> http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
> Did you mean 'Apache-2.0'?
> WARNING:  open-ended dependency on multi_json (>= 0) is not recommended
>   if multi_json is semantically versioned, use:
> add_runtime_dependency 'multi_json', '~> 0'
> WARNING:  See http://guides.rubygems.org/specification-reference/ for help
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AVRO-2468) Fix broken data interoperability on the Perl bindings

2019-08-20 Thread Fokko Driesprong (Jira)



 [ 
https://issues.apache.org/jira/browse/AVRO-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AVRO-2468:
---
Fix Version/s: 1.10.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Fix broken data interoperability on the Perl bindings
> -
>
> Key: AVRO-2468
> URL: https://issues.apache.org/jira/browse/AVRO-2468
> Project: Apache Avro
>  Issue Type: Bug
>  Components: interop, perl
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Critical
> Fix For: 1.10.0
>
>
> I found some data interop problems on the Perl bindings.
> 1. They fail to parse a schema if there's an array/map/union which contains 
> named types with a simple (not fully-qualified) name in it. For example, they 
> can't parse {{share/test/schemas/interop.avsc}} or 
> {{share/schemas/org/apache/avro/data/Json.avsc}}, because they have a named 
> type called "Node" or "Json" respectively in arrays/maps. This seems because 
> the parser doesn't take namespace into consideration in parsing 
> array/map/union.
> {code}
> $ cd lang/perl
> $ perl -Ilib -de 1
> (snip)
>   DB<1> open FH, '../../share/test/schemas/interop.avsc'; local $/ = undef; 
> $s = ; close FH; print $s
> {"type": "record", "name":"Interop", "namespace": "org.apache.avro",
>   "fields": [
>   {"name": "intField", "type": "int"},
>   {"name": "longField", "type": "long"},
>   {"name": "stringField", "type": "string"},
>   {"name": "boolField", "type": "boolean"},
>   {"name": "floatField", "type": "float"},
>   {"name": "doubleField", "type": "double"},
>   {"name": "bytesField", "type": "bytes"},
>   {"name": "nullField", "type": "null"},
>   {"name": "arrayField", "type": {"type": "array", "items": "double"}},
>   {"name": "mapField", "type":
>{"type": "map", "values":
> {"type": "record", "name": "Foo",
>  "fields": [{"name": "label", "type": "string"}]}}},
>   {"name": "unionField", "type":
>["boolean", "double", {"type": "array", "items": "bytes"}]},
>   {"name": "enumField", "type":
>{"type": "enum", "name": "Kind", "symbols": ["A","B","C"]}},
>   {"name": "fixedField", "type":
>{"type": "fixed", "name": "MD5", "size": 16}},
>   {"name": "recordField", "type":
>{"type": "record", "name": "Node",
> "fields": [
> {"name": "label", "type": "string"},
> {"name": "children", "type": {"type": "array", "items": 
> "Node"}}]}}
>   ]
> }
>   DB<2> use Avro::Schema; Avro::Schema->parse($s)
> Not a primitive type Node at lib/Avro/Schema.pm line 257.
> {code}
> 2. They encode the size for a fixed type as a string rather than a number, so 
> other language bindings fail to parse it.
> {code}
> $ cd lang/perl 
> $ perl -Ilib -de 1
> (snip)
>   DB<1> use Avro::Schema; $s = Avro::Schema->parse('{"type": "fixed", "size": 
> 16, "name": "md5"}')
>   DB<2> open($fh, '>/tmp/output')
>   DB<3> use Avro::DataFileWriter; $w = Avro::DataFileWriter->new(fh => $fh, 
> writer_schema => $s)
>   DB<4> $w->print('0123456789abcdef')
>   DB<5> $w->close
> {code}
> {code}
> $ ipython
> (snip)
> In [1]: from avro.datafile import DataFileReader
> In [2]: from avro.io import DatumReader
> In [3]: DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> ---
> AvroException Traceback (most recent call last)
>  in ()
> > 1 DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> /home/sekikn/repo/avro/lang/py/src/avro/datafile.pyc in __init__(self, 
> reader, datum_reader)
> 255 # get ready to read
> 256 self._block_count = 0
> --> 257 self.datum_reader.writers_schema = 
> schema.parse(self.get_meta(SCHEMA_KEY))
> 258 
> 259   def __enter__(self):
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in parse(json_string)
> 984 
> 985   # construct the Avro Schema object
> --> 986   return make_avsc_object(json_data, names)
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in 
> make_avsc_object(json_data, names)
> 931   scale = 0 if json_data.get('scale') is None else 
> json_data.get('scale')
> 932   return FixedDecimalSchema(size, name, precision, scale, 
> namespace, names, other_props)
> --> 933 return FixedSchema(name, namespace, size, names, other_props)
> 934   elif type == 'enum':
> 935 symbols = json_data.get('symbols')
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in __init__(self, name, 
> namespace, size, names, other_props)
> 482 if not isinstance(size, int) or size < 0:
> 483   fail_msg = 'Fixed Schema requires a valid positive

[jira] [Created] (AVRO-2511) Avro Java DataFileWriter Flush() does not flush the buffer to disk

2019-08-20 Thread Asen Milchev Kolev (Jira)

Asen Milchev Kolev created AVRO-2511:


 Summary: Avro Java DataFileWriter Flush() does not flush the 
buffer to disk
 Key: AVRO-2511
 URL: https://issues.apache.org/jira/browse/AVRO-2511
 Project: Apache Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.9.0
Reporter: Asen Milchev Kolev


If you try to use flush() with output stream instead of a file, the buffer is 
not flushed to disk. Here is an example how I'm using it and there is no chance 
to see data flushed to disk! Is that by design or it is a bug? I really need 
this in order to determine file size and create a new one when max file size is 
reached!
{code:java}

try (DataFileWriter dataFileWriter = new DataFileWriter<>(new 
GenericDatumWriter<>());                     DataFileStream 
dataFileStreamReader = new DataFileStream<>(is, new 
GenericDatumReader()))                {                    
dataFileWriter.setFlushOnEveryBlock(true);                    
FSDataOutputStream hdfsOutputStream = null;                    
dataFileWriter.setCodec(codecFactory);                    Schema schema = 
dataFileStreamReader.getSchema();                    if 
(fileMode.equals(FileMode.APPEND))                    {                        
FileContext fc = FileContext.getFileContext(hdfsConfiguration);                 
       hdfsOutputStream = fileSystem.append(hdfsPath);                        
dataFileWriter.appendTo(new AvroFSInput(fc, hdfsPath), hdfsOutputStream);       
             }                    else                    {                     
   hdfsOutputStream = fileSystem.create(hdfsPath);                        
fileManager.setCreationTime(hdfsPath);                        
dataFileWriter.create(schema, hdfsOutputStream);                    }
                    GenericRecord genericRecord = null;                    
while (dataFileStreamReader.hasNext())                    {                     
   if (fileManager.isLimitsReached())                        {                  
          IOUtils.closeStream(dataFileWriter);                            
fileSystem.rename(hdfsPath, fileManager.getFinalPath(hdfsPath));                
            LOG.info("Avro write completed for {0}", hdfsPath.toString());      
                      hdfsPath = fileManager.getPath();                         
   storePaths.add(hdfsPath);                            hdfsOutputStream = 
fileSystem.create(hdfsPath);                            
fileManager.setCreationTime(hdfsPath);                            
dataFileWriter.setCodec(codecFactory);                            
dataFileWriter.create(schema, hdfsOutputStream);                            
LOG.info("Initiate Avro write to {0}", hdfsPath.toString());                    
    }                        genericRecord = 
dataFileStreamReader.next(genericRecord);                        
dataFileWriter.append(genericRecord); dataFileWriter.flush();// doesn't work at 
all when we are using streams and not directly files!                        
fileManager.updateEntryCount(hdfsPath);                    }
{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] Release Apache Avro 1.9.1 RC1

Re: [VOTE] Release Apache Avro 1.9.1 RC1

Re: [VOTE] Release Apache Avro 1.9.1 RC1

[VOTE] Release Apache Avro 1.9.1 RC1

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

[jira] [Assigned] (AVRO-1852) Make org.apache.avro.Schema serializable (java.io.Serializable)

[jira] [Resolved] (AVRO-2513) Install Perl packages over https

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

[jira] [Updated] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

[jira] [Created] (AVRO-2514) Making Avro serialization error message more actionable with schema name and field name

[jira] [Commented] (AVRO-2512) writerField is always null

[jira] [Commented] (AVRO-2513) Install Perl packages over https

[jira] [Commented] (AVRO-2512) writerField is always null

[jira] [Commented] (AVRO-2512) writerField is always null

[jira] [Commented] (AVRO-2513) Install Perl packages over https

[jira] [Commented] (AVRO-2513) Install Perl packages over https

[jira] [Created] (AVRO-2513) Install Perl packages over https

[jira] [Resolved] (AVRO-2491) Schema.createRecord allows to create non parseable Schemas

[jira] [Updated] (AVRO-2486) fwrite error ignored

[jira] [Resolved] (AVRO-2414) Fix flaky TestDataFileReader::getNumberOfOpenFileDescriptors on JDK11

[jira] [Created] (AVRO-2512) writerField is always null

[jira] [Updated] (AVRO-2511) Avro Java DataFileWriter Flush() does not flush the buffer to disk

[jira] [Commented] (AVRO-2468) Fix broken data interoperability on the Perl bindings

[jira] [Commented] (AVRO-2506) Fix warnings in creating the Ruby distribution

[jira] [Commented] (AVRO-2506) Fix warnings in creating the Ruby distribution

[jira] [Updated] (AVRO-2468) Fix broken data interoperability on the Perl bindings

[jira] [Created] (AVRO-2511) Avro Java DataFileWriter Flush() does not flush the buffer to disk

27 matches

Site Navigation

Mail list logo

Footer information