[
https://issues.apache.org/jira/browse/BEAM-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545871#comment-16545871
]
Ryan Williams commented on BEAM-2810:
-------------------------------------
Thanks [~barrywhart], it's been a breeze to work with fastavro, and it should
be usable (behind a flag) in an upcoming Beam release quite soon! Definitely
interested to keep the dialogue open as we depend on it more here.
With [the integration test merged|https://github.com/apache/beam/pull/5862], I
think we're close to declaring victory here. Outstanding items:
* would be nice to get the OSX-installation story smoothed out (BEAM-4749)
* after the integration test has been running successfully for a few weeks,
and maybe once a Beam release has gone out and we've seen real production use,
we can talk about making fastavro the default; I'm thinking I'll file a
separate JIRA for that when appropriate.
Since those have or will have separate JIRAs, seems like we can mark this
"Resolved"?
> Consider a faster Avro library in Python
> ----------------------------------------
>
> Key: BEAM-2810
> URL: https://issues.apache.org/jira/browse/BEAM-2810
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Eugene Kirpichov
> Assignee: Ryan Williams
> Priority: Major
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> https://stackoverflow.com/questions/45870789/bottleneck-on-data-source
> Seems like this job is reading Avro files (exported by BigQuery) at about 2
> MB/s.
> We use the standard Python "avro" library which is apparently known to be
> very slow (10x+ slower than Java)
> http://apache-avro.679487.n3.nabble.com/Avro-decode-very-slow-in-Python-td4034422.html,
> and there are alternatives e.g. https://pypi.python.org/pypi/fastavro/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
