[jira] [Created] (AVRO-2291) GenericData.Array not reusable after AVRO-2050

[Martin Jubelgas (JIRA)]

Martin Jubelgas created AVRO-2291:
-

 Summary: GenericData.Array not reusable after AVRO-2050
 Key: AVRO-2291
 URL: https://issues.apache.org/jira/browse/AVRO-2291
 Project: Apache Avro
  Issue Type: Improvement
Reporter: Martin Jubelgas


The fix for AVRO-2050 left the reusing functionality of GenericData.Array 
broken. By nulling all fields of the underlying array during 
GenericData.Array.clear(), there was no more data to be reused in 
GenericDatumReader.readArray().

I've already posted a pull request to alleviate this issue in a backward 
compatible matter as a possible solution. Comments welcome.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-1940) avrogencpp, scoped enums or c++11 enum class

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp updated AVRO-1940:
--
   Resolution: Fixed
Fix Version/s: 1.9.0
   Status: Resolved  (was: Patch Available)

1.9 is using c++11 and now uses enum classes.

> avrogencpp, scoped enums or c++11 enum class
> 
>
> Key: AVRO-1940
> URL: https://issues.apache.org/jira/browse/AVRO-1940
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: c++
>Reporter: Alexander Moriarty
>Assignee: Alexander Moriarty
>Priority: Major
> Fix For: 1.9.0
>
>
> It would be nice if avrogencpp could generated scoped enums.
> Either with C++11 enum class or just adding an additional namespace.
> This sort of relates to the issue I opened a few days ago: #AVRO-1936 in that 
> one way this has been gotten around in the existing AvrogencppTests.cc is by 
> generating one namespace per schema.
> This of course could be a breaking change depending how it is implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2286) DataFileReader leaks file descriptor on exception during construction

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2286.
---
   Resolution: Fixed
Fix Version/s: 1.9.0

> DataFileReader leaks file descriptor on exception during construction
> -
>
> Key: AVRO-2286
> URL: https://issues.apache.org/jira/browse/AVRO-2286
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Holger Fleischmann
>Assignee: Martin Jubelgas
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: DataFileReaderCloseBug.java
>
>
> The constructor {{DataFileReader(File, DatumReader)}} does not close the 
> already opened file again when it throws an exception. This leads to a leaked 
> file descriptor and especially to not being able to delete the still open 
> invalid file then (at least on Windows).
> This case is important when Avro files may be truncated on write, e.g. due to 
> a hard shutdown, and the application must be able to recover from such a 
> situation on the next restart.
> Please see the attached file {{DataFileReaderCloseBug.java}} to reproduce the 
> problem.
> I did not find that problem in Jira/mailing lists. Thanks for your help in 
> advance :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2286) DataFileReader leaks file descriptor on exception during construction

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726215#comment-16726215
 ] 

ASF subversion and git services commented on AVRO-2286:
---

Commit a262a107ddd403e3ac5a773c735110a6565104de in avro's branch 
refs/heads/master from Martin Jubelgas
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=a262a10 ]

Fix for AVRO-2286 and regression test (#406)



> DataFileReader leaks file descriptor on exception during construction
> -
>
> Key: AVRO-2286
> URL: https://issues.apache.org/jira/browse/AVRO-2286
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Holger Fleischmann
>Assignee: Martin Jubelgas
>Priority: Major
> Attachments: DataFileReaderCloseBug.java
>
>
> The constructor {{DataFileReader(File, DatumReader)}} does not close the 
> already opened file again when it throws an exception. This leads to a leaked 
> file descriptor and especially to not being able to delete the still open 
> invalid file then (at least on Windows).
> This case is important when Avro files may be truncated on write, e.g. due to 
> a hard shutdown, and the application must be able to recover from such a 
> situation on the next restart.
> Please see the attached file {{DataFileReaderCloseBug.java}} to reproduce the 
> problem.
> I did not find that problem in Jira/mailing lists. Thanks for your help in 
> advance :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-1741) Python 3 code should check if avro.codec is None before decoding

[Anthony Hsu (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726175#comment-16726175
 ] 

Anthony Hsu commented on AVRO-1741:
---

This Jira can be resolved. The fix is committed 
[here|https://github.com/apache/avro/commit/191badb80281498611ffcbc0e79b9b020a3d5096#diff-4c24d0cfd2675026f0b87b9bfcaac54e]
 and present in avro-python3 1.8.2.

> Python 3 code should check if avro.codec is None before decoding
> 
>
> Key: AVRO-1741
> URL: https://issues.apache.org/jira/browse/AVRO-1741
> Project: Apache Avro
>  Issue Type: Bug
>Reporter: Matthew Hayes
>Assignee: Matthew Hayes
>Priority: Major
> Fix For: 1.7.7
>
> Attachments: AVRO-1741.diff
>
>
> I'm getting the error below when trying to read an Avro file in Python 3.  
> FWIW, I generated this Avro file from Pig.  I don't know if that makes a 
> difference.
> {code}
> avro.datafile.DataFileReader(open("test.avro","rb"),avro.io.DatumReader())
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/Users/mhayes/.pyenv/versions/3.4.2/lib/python3.4/site-packages/avro/datafile.py",
>  line 352, in __init__
> self.codec = self.GetMeta('avro.codec').decode('utf-8')
> AttributeError: 'NoneType' object has no attribute 'decode'
> {code}
> It appears the problem is in the following code.  It should check if GetMeta 
> returns None before trying to decode it.  The Python 2 code doesn't have this 
> problem because it is already decoded.
> {code}
> # ensure codec is valid
> self.codec = self.GetMeta('avro.codec').decode('utf-8')
> if self.codec is None:
>   self.codec = "null"
> if self.codec not in VALID_CODECS:
>   raise DataFileException('Unknown codec: %s.' % self.codec)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Modern PHP support

[Driesprong, Fokko]

Hi Gildas,

This would be very welcome. Currently, I'm working on getting the CI to
work with a newer base image . But
the tests only pass against a specific version of PHP 5.6. Let me know if
there is anything that I can do.

You could use git blame
 to
check who worked on the PHP stuff, but most of it has been migrated from
SVN so a lot of history is lost. If you feel confident in taking ownership
of the PHP, please do! :-)

Cheers, Fokko

Op di 18 dec. 2018 om 17:26 schreef Gildas Quéméner <
gildas.queme...@gmail.com>:

> Hello Daniel,
>
> Thanks for your answer.
>
> I will prepare a pull request in the next days (basically more or less with
> the content of https://gitlab.com/Jaumo/avro-php/).
>
> The most important things to do in order to make this package usable by any
> php projects out there is:
>
> 1) To synchronize a separate read-only repository with the content of
> https://github.com/apache/avro/tree/master/lang/php/lib (I'm looking at
> various solution to do so) => required by 3).
> 2) To setup a serious CI/CD pipeline which jobs will be to run the tests
> (against several php environments), check the code consistency and
> synchronize the read only repository.
> 3) To register the php package on packagist.org
>
> I would gladly take responsibility for these actions as I have experience
> in this area and will directly benefit from them.
>
> Problem is that this would require some administrative permission on the
> org/repository and I would probably need a referent at the ASF to know how
> things are done in your organization.
> Is there already someone in charge of the PHP implementation I could
> contact directly?
> If you have some apply process to become an official maintainer as well,
> please let me know.
>
> Cheers.
>
> *Gildas Quéméner*
>
>
> On Tue, Dec 18, 2018 at 8:11 PM Daniel Kulp  wrote:
>
> > We would definitely welcome any php contributions.   If you have any
> > specific fixes or updates or anything, PR’s would be more than welcome.
> >  It would be great if someone with some PHP expertise could take some
> > ownership in collecting various fixes/enhancements from various locations
> > and getting pull requests submitted for review.   That would be awesome.
> >
> > At this point, we won’t be splitting out the various languages to
> separate
> > repositories.   We have discussed it in the past and we MAY do it in the
> > future, but right now we’re trying to concentrate on getting 1.9 out (and
> > then likely a patch or two follow up) and splitting the repo right now
> > would be too disruptive to that process.
> >
> >
> > Dan
> >
> >
> > > On Dec 18, 2018, at 1:26 AM, Gildas Quéméner <
> gildas.queme...@gmail.com>
> > wrote:
> > >
> > > Hello,
> > >
> > > We've been working for a few months on using Avro encoded messages
> > between
> > > our services in my company.
> > >
> > > Our backend heavily relies on PHP, whereas some orbiting services are
> > > written in Java/Kotlin/Python.
> > > At first glance, the lake of official support for PHP looked like a big
> > > no-go for us.
> > >
> > > I know that there is an official library, embedded in the avro library
> at
> > > https://github.com/apache/avro/tree/master/lang/php/lib.
> > >
> > > However this library suffers from blocking issues:
> > > - It is not available through the de facto standard Composer dependency
> > > manager
> > > - It is not tested
> > > - It does not rely on autoloading PSR
> > > - It uses global namespace
> > > - It uses unknown php functions
> > > - It lacks some spec features (support for logicalType attribute for
> > > instance)
> > > - It is buggy
> > >
> > > Despite these issues, latest commit is 1 year old (about a typo), and
> the
> > > one before that is almost 3 years old. Thus, I think it is safe to say
> > that
> > > this library is not maintained anymore.
> > >
> > > Many organizations have realized such issue and have created their own
> > fork
> > > of the library, none of them being thoroughly maintained and having the
> > > same level of bug fixing.
> > >
> > > Here are a few examples:
> > > - https://github.com/wikimedia/avro-php
> > > - https://github.com/flix-tech/avro-php
> > > - https://github.com/researchgate/avro-php
> > > - https://gitlab.com/Jaumo/avro-php (I am the maintainer of this one)
> > >
> > > I have tried to contribute to Wikimedia's fork, but they also seem to
> > have
> > > dropped support (see activity in
> > > https://gerrit.wikimedia.org/r/q/status:open+project:avro-php). The
> > awesome
> > > thing is that they have brought a test suite (that we are reusing in
> our
> > > fork)!
> > >
> > > Flix-tech is doing an awesome job at providing a Schema Registry API
> > client
> > > as well as an Avro Serializer/Deserializer using their Schema Registry
> > > client.
> > >
> > > There are great initiatives out there to modernize this library, but
> none
> > > of 

[jira] [Created] (AVRO-2290) TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes breaks on Java 11

[JIRA]

Ismaël Mejía created AVRO-2290:
--

 Summary: TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes 
breaks on Java 11
 Key: AVRO-2290
 URL: https://issues.apache.org/jira/browse/AVRO-2290
 Project: Apache Avro
  Issue Type: Sub-task
Reporter: Ismaël Mejía


Java 11 has nano-second precision so the test breaks, the test should now 
detect this and adapt accordingly.

{code}
[ERROR] testRecordWithJsr310LogicalTypes(org.apache.avro.specific.TestSpecificLo
gicalTypes)  Time elapsed: 0.24 s  <<< FAILURE!
java.lang.AssertionError: Should match written record expected:<{"b": true, "i32
": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018-12-20, "t": 
15:02:53.535103, "ts": 2018-12-20T14:02:53.535127Z, "dec": 123.45}> but was:<{"b
": true, "i32": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018
-12-20, "t": 15:02:53.535, "ts": 2018-12-20T14:02:53.535Z, "dec": 123.45}>
at org.apache.avro.specific.TestSpecificLogicalTypes.testRecordWithJsr31
0LogicalTypes(TestSpecificLogicalTypes.java:132)

[ERROR] testAbilityToReadJodaRecordWrittenAsJsr310Record(org.apache.avro.specifi
c.TestSpecificLogicalTypes)  Time elapsed: 0.005 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: is "15:02:53.639158"
 but: was "15:02:53.639"
at org.apache.avro.specific.TestSpecificLogicalTypes.testAbilityToReadJo
daRecordWrittenAsJsr310Record(TestSpecificLogicalTypes.java:204)
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2288) Trevni’s OutputBuffer signature clashes with new ByteArrayOutputStream#writeBytes method

[JIRA]

 [ 
https://issues.apache.org/jira/browse/AVRO-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated AVRO-2288:
---
Issue Type: Sub-task  (was: Bug)
Parent: AVRO-2289

> Trevni’s OutputBuffer signature clashes with new 
> ByteArrayOutputStream#writeBytes method
> 
>
> Key: AVRO-2288
> URL: https://issues.apache.org/jira/browse/AVRO-2288
> Project: Apache Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> Java 11 added the new writeBytes method to ByteArrayOutputStream that clashes 
> with the pre-existing one. This can be fixed just by adjusting to the parent 
> signature as well as other methods to not throw Exceptions. The impact should 
> be minimal because this is a package private class.
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile 
> (default-compile) on project trevni-core: Compilation failure
> [ERROR] 
> /home/ismael/workspace/avro05/lang/java/trevni/core/src/main/java/org/apache/trevni/OutputBuffer.java:[102,15]
>  writeBytes(byte[]) in org.apache.trevni.OutputBuffer cannot override 
> writeBytes(byte[]) in java.io.ByteArrayOutputStream
> [ERROR]   overridden method does not throw java.io.IOException



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2289) Support Java 11

[JIRA]

Ismaël Mejía created AVRO-2289:
--

 Summary: Support Java 11
 Key: AVRO-2289
 URL: https://issues.apache.org/jira/browse/AVRO-2289
 Project: Apache Avro
  Issue Type: Improvement
  Components: java
Reporter: Ismaël Mejía


The goal of this issue is to track issues or needed changes to support Avro to 
compile and pass all tests using only JDK 11.

Java Module System is not part of the goals of this ticket and should be 
addressed later on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2288) Trevni’s OutputBuffer signature clashes with new ByteArrayOutputStream#writeBytes method

[JIRA]

Ismaël Mejía created AVRO-2288:
--

 Summary: Trevni’s OutputBuffer signature clashes with new 
ByteArrayOutputStream#writeBytes method
 Key: AVRO-2288
 URL: https://issues.apache.org/jira/browse/AVRO-2288
 Project: Apache Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.8.2
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


Java 11 added the new writeBytes method to ByteArrayOutputStream that clashes 
with the pre-existing one. This can be fixed just by adjusting to the parent 
signature as well as other methods to not throw Exceptions. The impact should 
be minimal because this is a package private class.

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) 
on project trevni-core: Compilation failure
[ERROR] 
/home/ismael/workspace/avro05/lang/java/trevni/core/src/main/java/org/apache/trevni/OutputBuffer.java:[102,15]
 writeBytes(byte[]) in org.apache.trevni.OutputBuffer cannot override 
writeBytes(byte[]) in java.io.ByteArrayOutputStream
[ERROR]   overridden method does not throw java.io.IOException




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2286) DataFileReader leaks file descriptor on exception during construction

[Holger Fleischmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725721#comment-16725721
 ] 

Holger Fleischmann commented on AVRO-2286:
--

Great, thanks! :)

> DataFileReader leaks file descriptor on exception during construction
> -
>
> Key: AVRO-2286
> URL: https://issues.apache.org/jira/browse/AVRO-2286
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Holger Fleischmann
>Assignee: Martin Jubelgas
>Priority: Major
> Attachments: DataFileReaderCloseBug.java
>
>
> The constructor {{DataFileReader(File, DatumReader)}} does not close the 
> already opened file again when it throws an exception. This leads to a leaked 
> file descriptor and especially to not being able to delete the still open 
> invalid file then (at least on Windows).
> This case is important when Avro files may be truncated on write, e.g. due to 
> a hard shutdown, and the application must be able to recover from such a 
> situation on the next restart.
> Please see the attached file {{DataFileReaderCloseBug.java}} to reproduce the 
> problem.
> I did not find that problem in Jira/mailing lists. Thanks for your help in 
> advance :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)