[jira] [Updated] (ARROW-4262) [Website] Blog post to give preview into using R and Arrow with Apache Spark
[ https://issues.apache.org/jira/browse/ARROW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-4262: -- Labels: pull-request-available (was: ) > [Website] Blog post to give preview into using R and Arrow with Apache Spark > > > Key: ARROW-4262 > URL: https://issues.apache.org/jira/browse/ARROW-4262 > Project: Apache Arrow > Issue Type: New Feature > Components: Website >Reporter: Wes McKinney >Assignee: Javier Luraschi >Priority: Major > Labels: pull-request-available > Fix For: 0.13.0 > > > See PR https://github.com/apache/arrow/pull/3001 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4283) Should RecordBatchStreamReader/Writer be AsyncIterable?
[
https://issues.apache.org/jira/browse/ARROW-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747321#comment-16747321
]
Paul Taylor commented on ARROW-4283:
[~pitrou] Thanks for the feedback.
I want to clarify: my Python skills aren't sharp, I'm not familiar with the
pyarrow API or Python's asyncio/async-iterable primitives, so filter my
comments through the lens of a beginner.
The little experience I do have is using the RecordBatchStreamReader to read
from stdin (via {{sys.stdin.buffer}}) and named file descriptors (via
{{os.fdopen()}}). Since Python's so friendly (and I have no idea how the Python
IO primitives work), I thought maybe I could pass aiohttp's {{Request.stream}}
to the RecordBatchStreamReader constructor, and quickly learned that no, I
can't ;).
In the JS implementation we have two main entry points for reading RecordBatch
streams:
# a static
[{{RecordBatchReader.from(source)}}|https://github.com/apache/arrow/blob/cc1ce6194b905768b1a6d9f0e209270f62dc558a/js/src/ipc/reader.ts#L142],
which accepts heterogeneous source types and returns a RecordBatchReader for
the underlying Arrow type (file, stream, or JSON) and conforms to sync/async
semantics of the source input type
# methods that create [through/transform
streams|https://github.com/apache/arrow/blob/cc1ce6194b905768b1a6d9f0e209270f62dc558a/js/bin/file-to-stream.js#L33]
from the RecordBatchReader and RecordBatchWriter, for use with node's native
stream primitives
Each link in the streaming pipeline is a sort of transform stream, and a
significant amount of effort went into supporting all the different
node/browser IO primitives, so I understand if that's too much to ask at this
point.
As an alternative, would it be possible to add a method that accepts a Python
byte stream, and returns a zero-copy AsyncIterable of RecordBatches? Or maybe
add an an example in the
[python/ipc|https://arrow.apache.org/docs/python/ipc.html#writing-and-reading-streams]
docs page of how to do that?
> Should RecordBatchStreamReader/Writer be AsyncIterable?
> ---
>
> Key: ARROW-4283
> URL: https://issues.apache.org/jira/browse/ARROW-4283
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Paul Taylor
>Priority: Minor
> Fix For: 0.13.0
>
>
> Filing this issue after a discussion today with [~xhochy] about how to
> implement streaming pyarrow http services. I had attempted to use both Flask
> and [aiohttp|https://aiohttp.readthedocs.io/en/stable/streams.html]'s
> streaming interfaces because they seemed familiar, but no dice. I have no
> idea how hard this would be to add -- supporting all the asynciterable
> primitives in JS was non-trivial.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-4301) [Java][Gandiva] Maven snapshot version update does not seem to update Gandiva submodule
[ https://issues.apache.org/jira/browse/ARROW-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-4301: Summary: [Java][Gandiva] Maven snapshot version update does not seem to update Gandiva submodule (was: [Java] Maven snapshot version update does not seem to update Gandiva submodule) > [Java][Gandiva] Maven snapshot version update does not seem to update Gandiva > submodule > --- > > Key: ARROW-4301 > URL: https://issues.apache.org/jira/browse/ARROW-4301 > Project: Apache Arrow > Issue Type: Bug > Components: Gandiva, Java >Reporter: Wes McKinney >Priority: Major > Fix For: 0.13.0 > > > See > https://github.com/apache/arrow/commit/a486db8c1476be1165981c4fe22996639da8e550. > This is breaking the build so I'm going to patch manually -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-4301) [Java] Maven snapshot version update does not seem to update Gandiva submodule
[ https://issues.apache.org/jira/browse/ARROW-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-4301: Component/s: Gandiva > [Java] Maven snapshot version update does not seem to update Gandiva submodule > -- > > Key: ARROW-4301 > URL: https://issues.apache.org/jira/browse/ARROW-4301 > Project: Apache Arrow > Issue Type: Bug > Components: Gandiva, Java >Reporter: Wes McKinney >Priority: Major > Fix For: 0.13.0 > > > See > https://github.com/apache/arrow/commit/a486db8c1476be1165981c4fe22996639da8e550. > This is breaking the build so I'm going to patch manually -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-4301) [Java] Maven snapshot version update does not seem to update Gandiva submodule
Wes McKinney created ARROW-4301: --- Summary: [Java] Maven snapshot version update does not seem to update Gandiva submodule Key: ARROW-4301 URL: https://issues.apache.org/jira/browse/ARROW-4301 Project: Apache Arrow Issue Type: Bug Components: Java Reporter: Wes McKinney Fix For: 0.13.0 See https://github.com/apache/arrow/commit/a486db8c1476be1165981c4fe22996639da8e550. This is breaking the build so I'm going to patch manually -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-4300) [C++] Restore apache-arrow Homebrew recipe and define process for maintaining and updating for releases
Wes McKinney created ARROW-4300: --- Summary: [C++] Restore apache-arrow Homebrew recipe and define process for maintaining and updating for releases Key: ARROW-4300 URL: https://issues.apache.org/jira/browse/ARROW-4300 Project: Apache Arrow Issue Type: New Feature Components: C++ Reporter: Wes McKinney Fix For: 0.13.0 As far as I know, the Apache Arrow community has not been maintaining https://github.com/Homebrew/homebrew-core/blob/9a4c569e7c6e63cb1171c2b2f209de2108a39eb7/Formula/apache-arrow.rb It was just dropped from Homebrew for lack of maintenance. Some installation targets may depend on Arrow from Homebrew, so we should include updating Homebrew in our release management process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4245) [Rust] Add Rustdoc header to each source file
[ https://issues.apache.org/jira/browse/ARROW-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747153#comment-16747153 ] Andy Grove commented on ARROW-4245: --- [~paulkernfeld] This is exactly what I had in mind :) I look forward to seeing the PR > [Rust] Add Rustdoc header to each source file > - > > Key: ARROW-4245 > URL: https://issues.apache.org/jira/browse/ARROW-4245 > Project: Apache Arrow > Issue Type: Improvement > Components: Rust >Reporter: Andy Grove >Priority: Minor > Fix For: 0.13.0 > > > Each source file should contain a rustdoc header giving a summary of the > contents of the file. This will make the documentation hosted on docs.rs much > more useful. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4245) [Rust] Add Rustdoc header to each source file
[ https://issues.apache.org/jira/browse/ARROW-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747134#comment-16747134 ] Paul Kernfeld commented on ARROW-4245: -- I'm interested in trying to do this. Could someone take a look at [these example docs I wrote|https://github.com/apache/arrow/compare/master...paulkernfeld:arrow-4245-add-rustdoc-headers] to see if I'm going in the right direction? If they look good, I'm happy to add more. I deliberately added a lot of examples in the docs because I find that I'm personally a lot more likely to engage with a project if it has lots of examples. > [Rust] Add Rustdoc header to each source file > - > > Key: ARROW-4245 > URL: https://issues.apache.org/jira/browse/ARROW-4245 > Project: Apache Arrow > Issue Type: Improvement > Components: Rust >Reporter: Andy Grove >Priority: Minor > Fix For: 0.13.0 > > > Each source file should contain a rustdoc header giving a summary of the > contents of the file. This will make the documentation hosted on docs.rs much > more useful. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4245) [Rust] Add Rustdoc header to each source file
[
https://issues.apache.org/jira/browse/ARROW-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747140#comment-16747140
]
Uwe L. Korn commented on ARROW-4245:
[~paulkernfeld] To get better feedback, I recommend to already open a PR on
GitHub and prefixing it with {{WIP}} (work in progress). This gives us the
better review tools while everyone is aware that the work is not done yet.
> [Rust] Add Rustdoc header to each source file
> -
>
> Key: ARROW-4245
> URL: https://issues.apache.org/jira/browse/ARROW-4245
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust
>Reporter: Andy Grove
>Priority: Minor
> Fix For: 0.13.0
>
>
> Each source file should contain a rustdoc header giving a summary of the
> contents of the file. This will make the documentation hosted on docs.rs much
> more useful.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-4299) [Ruby] Depend on the same version as Red Arrow
[
https://issues.apache.org/jira/browse/ARROW-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747273#comment-16747273
]
Kouhei Sutou commented on ARROW-4299:
-
[~shiro615] Could you work on this?
> [Ruby] Depend on the same version as Red Arrow
> --
>
> Key: ARROW-4299
> URL: https://issues.apache.org/jira/browse/ARROW-4299
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Ruby
>Reporter: Kouhei Sutou
>Assignee: Yosuke Shiro
>Priority: Major
>
> Red Arrow CUDA, Red Plasma, Red Gandiva and Red Parquet depend on Red Arrow.
> But they don't specify Red Arrow version.
> They don't work well without the same Red Arrow version. So they should
> depend on the same version as Red Arrow.
>
> We will be able to implement with the following change:
>
> {code:ruby}
> spec.add_runtime_dependency("red-arrow", "= #{spec.version}")
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-4295) [Plasma] Incorrect log message when evicting objects
[
https://issues.apache.org/jira/browse/ARROW-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Moritz resolved ARROW-4295.
---
Resolution: Fixed
Issue resolved by pull request 3433
[https://github.com/apache/arrow/pull/3433]
> [Plasma] Incorrect log message when evicting objects
>
>
> Key: ARROW-4295
> URL: https://issues.apache.org/jira/browse/ARROW-4295
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Plasma (C++)
>Affects Versions: 0.11.1
>Reporter: Anurag Khandelwal
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When Plasma evicts objects on running out of memory, it prints log messages
> of the form:
> {quote}There is not enough space to create this object, so evicting x objects
> to free up y bytes. The number of bytes in use (before this eviction) is
> z.{quote}
> However, the reported number of bytes in use (before this eviction) actually
> reports the number of bytes *after* the eviction. A straightforward fix is to
> simply replace z with (y+z).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-4291) [Dev] Support selecting features in release scripts
[ https://issues.apache.org/jira/browse/ARROW-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kouhei Sutou resolved ARROW-4291. - Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request 3430 [https://github.com/apache/arrow/pull/3430] > [Dev] Support selecting features in release scripts > --- > > Key: ARROW-4291 > URL: https://issues.apache.org/jira/browse/ARROW-4291 > Project: Apache Arrow > Issue Type: New Feature > Components: Developer Tools, Packaging >Reporter: Uwe L. Korn >Assignee: Uwe L. Korn >Priority: Major > Labels: pull-request-available > Fix For: 0.13.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Sometimes not all components can be verified on a system. We should provide > some environment variables to exclude them to proceed to the next step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-4295) [Plasma] Incorrect log message when evicting objects
[
https://issues.apache.org/jira/browse/ARROW-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Moritz reassigned ARROW-4295:
-
Assignee: Anurag Khandelwal
> [Plasma] Incorrect log message when evicting objects
>
>
> Key: ARROW-4295
> URL: https://issues.apache.org/jira/browse/ARROW-4295
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Plasma (C++)
>Affects Versions: 0.11.1
>Reporter: Anurag Khandelwal
>Assignee: Anurag Khandelwal
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.13.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When Plasma evicts objects on running out of memory, it prints log messages
> of the form:
> {quote}There is not enough space to create this object, so evicting x objects
> to free up y bytes. The number of bytes in use (before this eviction) is
> z.{quote}
> However, the reported number of bytes in use (before this eviction) actually
> reports the number of bytes *after* the eviction. A straightforward fix is to
> simply replace z with (y+z).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3204) [R] Release to CRAN
[ https://issues.apache.org/jira/browse/ARROW-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747277#comment-16747277 ] Jeroen commented on ARROW-3204: --- The Homebrew lead maintainer has dropped arrow from homebrew because it isn't building: https://github.com/Homebrew/homebrew-core/commit/6cd7f61d2d07d5fd9f8747ea0f620169cc8a2434 . So this further complicates things for R bindings on MacOS. I am guessing that the build broke because the 0.11 tarball has been removed from the apache website, even though 0.12 hasn't even been announced yet? [https://www.apache.org/dist/arrow/] > [R] Release to CRAN > --- > > Key: ARROW-3204 > URL: https://issues.apache.org/jira/browse/ARROW-3204 > Project: Apache Arrow > Issue Type: New Feature > Components: R >Reporter: James Lamb >Priority: Major > > As of [ARROW-1325|https://issues.apache.org/jira/browse/ARROW-1325], the > project contains a minimal working R package that takes advantage of R's > built-in support for calling C++ code and uses Rcpp for added support. > Though the exact public interface of the package hasn't been developed yet > (so this issue may be running before we walk), one major feature in its > development will be release to [CRAN|https://cran.r-project.org/], the > official package repository for the R language. > Completing this story would mean: > * Getting source code to state where [R CMD > CHECK|https://www.r-bloggers.com/how-i-learned-to-stop-worrying-and-love-r-cmd-check/] > passes with 0 ERRORS, 0 WARNINGS, and 0 NOTES > * Setting up build process for source and precompiled binary packages (see > [data.table|https://github.com/Rdatatable/data.table] and > [XGBoost|https://github.com/dmlc/xgboost] as examples of packages that do > this) > * Submission to CRAN and acceptance of the first release > > Distribution via CRAN would be a much more natural fit for most R users' > workflows than the current build-from-source workflow, and comes with all the > other benefits of package managers (e.g. version pegging, easy distribution > of platform-specific binaries). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-4297) [C++] Fix build and add CI configuration for 32-bit MSYS2
Wes McKinney created ARROW-4297: --- Summary: [C++] Fix build and add CI configuration for 32-bit MSYS2 Key: ARROW-4297 URL: https://issues.apache.org/jira/browse/ARROW-4297 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Wes McKinney Fix For: 0.13.0 see mailing list thread https://lists.apache.org/thread.html/0fc5f7ffe4eb4e4aea931f0b26902d27305cf98829fb5c4a7375f2ab@%3Cdev.arrow.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4298) [Java] Building Flight fails with OpenJDK 11
[
https://issues.apache.org/jira/browse/ARROW-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747203#comment-16747203
]
Jacques Nadeau commented on ARROW-4298:
---
I'm actually hopping in a plane momentarily and will be unavailable for the
next week. [~laurentgo]Goujon can you take a look?
> [Java] Building Flight fails with OpenJDK 11
>
>
> Key: ARROW-4298
> URL: https://issues.apache.org/jira/browse/ARROW-4298
> Project: Apache Arrow
> Issue Type: New Feature
> Components: FlightRPC, Java
>Affects Versions: 0.12.0
>Reporter: Uwe L. Korn
>Priority: Major
>
> Building flight fails with
> {code:java}
> [INFO] --- maven-compiler-plugin:3.6.2:compile (default-compile) @
> arrow-flight ---
> [INFO] Compiling 39 source files to
> /Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/classes
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/generated-sources/protobuf/org/apache/arrow/flight/impl/FlightServiceGrpc.java:[26,17]
> error: cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [INFO] 1 error{code}
> To fix this, I added the following dependency to {{flight/pom.xml}}:
> {code:java}
>
>javax.annotation
>javax.annotation-api
>1.3.2
> {code}
> This then passed the compile step but failed later with:
> {code:java}
> [INFO] --- maven-dependency-plugin:3.0.1:analyze-only (analyze) @
> arrow-flight ---
> [WARNING] Unused declared dependencies found:
> [WARNING] javax.annotation:javax.annotation-api:jar:1.3.2:compile
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze)
> on project arrow-flight: Dependency problems found -> [Help 1]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-4299) [Ruby] Depend on the same version as Red Arrow
Kouhei Sutou created ARROW-4299:
---
Summary: [Ruby] Depend on the same version as Red Arrow
Key: ARROW-4299
URL: https://issues.apache.org/jira/browse/ARROW-4299
Project: Apache Arrow
Issue Type: Improvement
Components: Ruby
Reporter: Kouhei Sutou
Assignee: Yosuke Shiro
Red Arrow CUDA, Red Plasma, Red Gandiva and Red Parquet depend on Red Arrow.
But they don't specify Red Arrow version.
They don't work well without the same Red Arrow version. So they should depend
on the same version as Red Arrow.
We will be able to implement with the following change:
{code:ruby}
spec.add_runtime_dependency("red-arrow", "= #{spec.version}")
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-4298) [Java] Building Flight fails with OpenJDK 11
[
https://issues.apache.org/jira/browse/ARROW-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747201#comment-16747201
]
Uwe L. Korn commented on ARROW-4298:
[~jacq...@dremio.com] Any indication on how to fix this would be appreciated.
> [Java] Building Flight fails with OpenJDK 11
>
>
> Key: ARROW-4298
> URL: https://issues.apache.org/jira/browse/ARROW-4298
> Project: Apache Arrow
> Issue Type: New Feature
> Components: FlightRPC, Java
>Affects Versions: 0.12.0
>Reporter: Uwe L. Korn
>Priority: Major
>
> Building flight fails with
> {code:java}
> [INFO] --- maven-compiler-plugin:3.6.2:compile (default-compile) @
> arrow-flight ---
> [INFO] Compiling 39 source files to
> /Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/classes
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/generated-sources/protobuf/org/apache/arrow/flight/impl/FlightServiceGrpc.java:[26,17]
> error: cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [INFO] 1 error{code}
> To fix this, I added the following dependency to {{flight/pom.xml}}:
> {code:java}
>
>javax.annotation
>javax.annotation-api
>1.3.2
> {code}
> This then passed the compile step but failed later with:
> {code:java}
> [INFO] --- maven-dependency-plugin:3.0.1:analyze-only (analyze) @
> arrow-flight ---
> [WARNING] Unused declared dependencies found:
> [WARNING] javax.annotation:javax.annotation-api:jar:1.3.2:compile
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze)
> on project arrow-flight: Dependency problems found -> [Help 1]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1299) [Doc] Publish nightly documentation against master somewhere
[ https://issues.apache.org/jira/browse/ARROW-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747226#comment-16747226 ] Uwe L. Korn commented on ARROW-1299: [~wesmckinn] Any ideas on which infrastructure we could do this? Should we concat INFRA for this or do we have other means? > [Doc] Publish nightly documentation against master somewhere > > > Key: ARROW-1299 > URL: https://issues.apache.org/jira/browse/ARROW-1299 > Project: Apache Arrow > Issue Type: Task > Components: Documentation >Reporter: Wes McKinney >Priority: Major > > This will help catch problems with the generated documentation prior to > release time, and also allow users to read the latest prose documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-4298) [Java] Building Flight fails with OpenJDK 11
Uwe L. Korn created ARROW-4298:
--
Summary: [Java] Building Flight fails with OpenJDK 11
Key: ARROW-4298
URL: https://issues.apache.org/jira/browse/ARROW-4298
Project: Apache Arrow
Issue Type: New Feature
Components: FlightRPC, Java
Affects Versions: 0.12.0
Reporter: Uwe L. Korn
Building flight fails with
{code:java}
[INFO] --- maven-compiler-plugin:3.6.2:compile (default-compile) @ arrow-flight
---
[INFO] Compiling 39 source files to
/Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/classes
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/uwe/Development/arrow-repos-1/arrow/java/flight/target/generated-sources/protobuf/org/apache/arrow/flight/impl/FlightServiceGrpc.java:[26,17]
error: cannot find symbol
symbol: class Generated
location: package javax.annotation
[INFO] 1 error{code}
To fix this, I added the following dependency to {{flight/pom.xml}}:
{code:java}
javax.annotation
javax.annotation-api
1.3.2
{code}
This then passed the compile step but failed later with:
{code:java}
[INFO] --- maven-dependency-plugin:3.0.1:analyze-only (analyze) @ arrow-flight
---
[WARNING] Unused declared dependencies found:
[WARNING] javax.annotation:javax.annotation-api:jar:1.3.2:compile
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze)
on project arrow-flight: Dependency problems found -> [Help 1]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (ARROW-3224) CI: Consider using azure pipelines?
[ https://issues.apache.org/jira/browse/ARROW-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-3224. --- > CI: Consider using azure pipelines? > --- > > Key: ARROW-3224 > URL: https://issues.apache.org/jira/browse/ARROW-3224 > Project: Apache Arrow > Issue Type: Bug > Components: Continuous Integration >Reporter: Krisztian Szucs >Priority: Minor > > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/ > It might simplify / speed up arrow's CI builds and/or the packaging. Does > anyone have an experience with it? > Pros based on the blog post: > - supports win, osx, linux > - 10 parallel builds -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-3238) [Python] Can't read pyarrow string columns in fastparquet
[
https://issues.apache.org/jira/browse/ARROW-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-3238.
---
> [Python] Can't read pyarrow string columns in fastparquet
> -
>
> Key: ARROW-3238
> URL: https://issues.apache.org/jira/browse/ARROW-3238
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Theo Walker
>Priority: Major
> Labels: parquet
>
> Writing really long strings from pyarrow causes exception in fastparquet read.
> {code:java}
> Traceback (most recent call last):
> File "/Users/twalker/repos/cloud-atlas/diag/right.py", line 47, in
> read_fastparquet()
> File "/Users/twalker/repos/cloud-atlas/diag/right.py", line 41, in
> read_fastparquet
> dff = pf.to_pandas(['A'])
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/api.py",
> line 426, in to_pandas
> index=index, assign=parts)
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/api.py",
> line 258, in read_row_group
> scheme=self.file_scheme)
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/core.py",
> line 344, in read_row_group
> cats, selfmade, assign=assign)
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/core.py",
> line 321, in read_row_group_arrays
> catdef=out.get(name+'-catdef', None))
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/core.py",
> line 235, in read_col
> skip_nulls, selfmade=selfmade)
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/core.py",
> line 99, in read_data_page
> raw_bytes = _read_page(f, header, metadata)
> File
> "/Users/twalker/anaconda/lib/python2.7/site-packages/fastparquet/core.py",
> line 31, in _read_page
> page_header.uncompressed_page_size)
> AssertionError: found 175532 raw bytes (expected 200026){code}
> If written with compression, it reports compression errors instead:
> {code:java}
> SNAPPY: snappy.UncompressError: Error while decompressing: invalid input
> GZIP: zlib.error: Error -3 while decompressing data: incorrect header
> check{code}
>
>
> Minimal code to reproduce:
> {code:java}
> import os
> import pandas as pd
> import pyarrow
> import pyarrow.parquet as arrow_pq
> from fastparquet import ParquetFile
> # data to generate
> ROW_LENGTH = 4 # decreasing below 32750ish eliminates exception
> N_ROWS = 10
> # file write params
> ROW_GROUP_SIZE = 5 # Lower numbers eliminate exception, but strange data is
> read (e.g. Nones)
> FILENAME = 'test.parquet'
> def write_arrow():
> df = pd.DataFrame({'A': ['A'*ROW_LENGTH for _ in range(N_ROWS)]})
> if os.path.isfile(FILENAME):
> os.remove(FILENAME)
> arrow_table = pyarrow.Table.from_pandas(df)
> arrow_pq.write_table(arrow_table,
> FILENAME,
> use_dictionary=False,
> compression='NONE',
> row_group_size=ROW_GROUP_SIZE)
> def read_arrow():
> print "arrow:"
> table2 = arrow_pq.read_table(FILENAME)
> print table2.to_pandas().head()
> def read_fastparquet():
> print "fastparquet:"
> pf = ParquetFile(FILENAME)
> dff = pf.to_pandas(['A'])
> print dff.head()
> write_arrow()
> read_arrow()
> read_fastparquet()
> {code}
> Versions:
> {code:java}
> fastparquet==0.1.6
> pyarrow==0.10.0
> pandas==0.22.0
> sys.version '2.7.15 |Anaconda custom (64-bit)| (default, May 1 2018,
> 18:37:05) \n[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]'{code}
> Also opened issue here: https://github.com/dask/fastparquet/issues/375
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2648) [Packaging] Follow up packaging tasks
[ https://issues.apache.org/jira/browse/ARROW-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-2648: Fix Version/s: 0.12.0 > [Packaging] Follow up packaging tasks > - > > Key: ARROW-2648 > URL: https://issues.apache.org/jira/browse/ARROW-2648 > Project: Apache Arrow > Issue Type: Task > Components: Packaging >Affects Versions: 0.10.0 >Reporter: Phillip Cloud >Assignee: Krisztian Szucs >Priority: Major > Fix For: 0.12.0 > > > We may want to split these out into separate JIRAs, but I want to get a list > of follow up tasks that are necessary for a 0.10.0 release: > - Uploading artifacts > - Downloading artifacts > - Signing/verification of packages > - Release management guide with new packaging workflow -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-3663) pyarrow install via pip3 fails with error no module named Cython
[
https://issues.apache.org/jira/browse/ARROW-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-3663.
---
> pyarrow install via pip3 fails with error no module named Cython
>
>
> Key: ARROW-3663
> URL: https://issues.apache.org/jira/browse/ARROW-3663
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.11.1
>Reporter: Rajshekhar K
>Assignee: Uwe L. Korn
>Priority: Trivial
>
> Hi Team,
>
> The issue is reproducible :
> # pip3 install pyarrow
> Fails installation with no module name Cython. Seems it's not mentioned in
> the requirements or something.
>
> {code:java}
> Downloading pyarrow-0.10.0.tar.gz (2.1MB): 2.1MB downloaded
> Running setup.py (path:/tmp/pip_build_root/pyarrow/setup.py) egg_info for
> package pyarrow
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pyarrow/setup.py", line 29, in
> from Cython.Distutils import build_ext as _build_ext
> ImportError: No module named 'Cython'
> Complete output from command python setup.py egg_info:
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pyarrow/setup.py", line 29, in
> from Cython.Distutils import build_ext as _build_ext
> ImportError: No module named 'Cython'
>
> Cleaning up...
> {code}
>
>
> Tested on Environment: ubuntu14.04
> Pip version:
> {noformat}
> pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.4){noformat}
>
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (ARROW-2067) [Python] "pip install" doesn't work from source tree
[
https://issues.apache.org/jira/browse/ARROW-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-2067.
---
> [Python] "pip install" doesn't work from source tree
>
>
> Key: ARROW-2067
> URL: https://issues.apache.org/jira/browse/ARROW-2067
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.8.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
>
> It seems that {{pip install .}} first copies the build dir into a temporary
> directory, and {{setuptools_scm}} then fails grabbing the git version from
> that location.
> AFAIR {{versioneer}} doesn't have that issue.
> {code:bash}
> $ pip install .
> Processing /home/antoine/arrow/python
> Complete output from command python setup.py egg_info:
> Traceback (most recent call last):
> File "", line 1, in
> File "/tmp/pip-v_mucrpj-build/setup.py", line 456, in
> url="https://arrow.apache.org/;
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/site-packages/setuptools/__init__.py",
> line 129, in setup
> return distutils.core.setup(**attrs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/distutils/core.py", line
> 108, in setup
> _setup_distribution = dist = klass(attrs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/site-packages/setuptools/dist.py",
> line 333, in __init__
> _Distribution.__init__(self, attrs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/distutils/dist.py", line
> 281, in __init__
> self.finalize_options()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/site-packages/setuptools/dist.py",
> line 476, in finalize_options
> ep.load()(self, ep.name, value)
> File
> "/tmp/pip-v_mucrpj-build/.eggs/setuptools_scm-1.15.7-py3.6.egg/setuptools_scm/integration.py",
> line 22, in version_keyword
> dist.metadata.version = get_version(**value)
> File
> "/tmp/pip-v_mucrpj-build/.eggs/setuptools_scm-1.15.7-py3.6.egg/setuptools_scm/__init__.py",
> line 119, in get_version
> parsed_version = _do_parse(root, parse)
> File
> "/tmp/pip-v_mucrpj-build/.eggs/setuptools_scm-1.15.7-py3.6.egg/setuptools_scm/__init__.py",
> line 97, in _do_parse
> "use git+https://github.com/user/proj.git#egg=proj; % root)
> LookupError: setuptools-scm was unable to detect version for
> '/tmp/pip-v_mucrpj-build'.
>
> Make sure you're either building from a fully intact git repository or
> PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout
> without the .git folder) don't contain the necessary metadata and will not
> work.
>
> For example, if you're using pip, instead of
> https://github.com/user/proj/archive/master.zip use
> git+https://github.com/user/proj.git#egg=proj
>
>
> Command "python setup.py egg_info" failed with error code 1 in
> /tmp/pip-v_mucrpj-build/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-1262) [Packaging] Packaging automation in arrow-dist
[ https://issues.apache.org/jira/browse/ARROW-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1262: Fix Version/s: 0.12.0 > [Packaging] Packaging automation in arrow-dist > -- > > Key: ARROW-1262 > URL: https://issues.apache.org/jira/browse/ARROW-1262 > Project: Apache Arrow > Issue Type: Task > Components: Packaging >Reporter: Wes McKinney >Priority: Major > Fix For: 0.12.0 > > > This JIRA is an umbrella JIRA for tasks to streamline our binary builds at > release time as much as possible. We may also be able to set up nightly > builds for testing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-3204) [R] Release to CRAN
[ https://issues.apache.org/jira/browse/ARROW-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747309#comment-16747309 ] Wes McKinney commented on ARROW-3204: - AFAIK no one from this community has been maintaining the Homebrew recipe. So that will obviously have to change so that updating the Homebrew recipe is part of the release management process. No one did anything wrong here I'm creating a sub-task about Homebrew and attaching it as a child of this issue > [R] Release to CRAN > --- > > Key: ARROW-3204 > URL: https://issues.apache.org/jira/browse/ARROW-3204 > Project: Apache Arrow > Issue Type: New Feature > Components: R >Reporter: James Lamb >Priority: Major > > As of [ARROW-1325|https://issues.apache.org/jira/browse/ARROW-1325], the > project contains a minimal working R package that takes advantage of R's > built-in support for calling C++ code and uses Rcpp for added support. > Though the exact public interface of the package hasn't been developed yet > (so this issue may be running before we walk), one major feature in its > development will be release to [CRAN|https://cran.r-project.org/], the > official package repository for the R language. > Completing this story would mean: > * Getting source code to state where [R CMD > CHECK|https://www.r-bloggers.com/how-i-learned-to-stop-worrying-and-love-r-cmd-check/] > passes with 0 ERRORS, 0 WARNINGS, and 0 NOTES > * Setting up build process for source and precompiled binary packages (see > [data.table|https://github.com/Rdatatable/data.table] and > [XGBoost|https://github.com/dmlc/xgboost] as examples of packages that do > this) > * Submission to CRAN and acceptance of the first release > > Distribution via CRAN would be a much more natural fit for most R users' > workflows than the current build-from-source workflow, and comes with all the > other benefits of package managers (e.g. version pegging, easy distribution > of platform-specific binaries). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ARROW-3204) [R] Release to CRAN
[ https://issues.apache.org/jira/browse/ARROW-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747309#comment-16747309 ] Wes McKinney edited comment on ARROW-3204 at 1/20/19 1:49 AM: -- AFAIK no one from this community has been maintaining the Homebrew recipe. So that will obviously have to change so that updating the Homebrew recipe is part of the release management process. No one did anything wrong here I'm creating a sub-task about Homebrew and attaching it as a child of this issue EDIT: opened ARROW-4300 was (Author: wesmckinn): AFAIK no one from this community has been maintaining the Homebrew recipe. So that will obviously have to change so that updating the Homebrew recipe is part of the release management process. No one did anything wrong here I'm creating a sub-task about Homebrew and attaching it as a child of this issue > [R] Release to CRAN > --- > > Key: ARROW-3204 > URL: https://issues.apache.org/jira/browse/ARROW-3204 > Project: Apache Arrow > Issue Type: New Feature > Components: R >Reporter: James Lamb >Priority: Major > > As of [ARROW-1325|https://issues.apache.org/jira/browse/ARROW-1325], the > project contains a minimal working R package that takes advantage of R's > built-in support for calling C++ code and uses Rcpp for added support. > Though the exact public interface of the package hasn't been developed yet > (so this issue may be running before we walk), one major feature in its > development will be release to [CRAN|https://cran.r-project.org/], the > official package repository for the R language. > Completing this story would mean: > * Getting source code to state where [R CMD > CHECK|https://www.r-bloggers.com/how-i-learned-to-stop-worrying-and-love-r-cmd-check/] > passes with 0 ERRORS, 0 WARNINGS, and 0 NOTES > * Setting up build process for source and precompiled binary packages (see > [data.table|https://github.com/Rdatatable/data.table] and > [XGBoost|https://github.com/dmlc/xgboost] as examples of packages that do > this) > * Submission to CRAN and acceptance of the first release > > Distribution via CRAN would be a much more natural fit for most R users' > workflows than the current build-from-source workflow, and comes with all the > other benefits of package managers (e.g. version pegging, easy distribution > of platform-specific binaries). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-3204) [R] Release to CRAN
[ https://issues.apache.org/jira/browse/ARROW-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747314#comment-16747314 ] Wes McKinney commented on ARROW-3204: - I'll keep attaching child issues to this issue that are required (or related, like setting up CI for Windows) as part of the testing and packaging process. If there is anything necessary that is not in the issue tracker already please create new issues > [R] Release to CRAN > --- > > Key: ARROW-3204 > URL: https://issues.apache.org/jira/browse/ARROW-3204 > Project: Apache Arrow > Issue Type: New Feature > Components: R >Reporter: James Lamb >Priority: Major > > As of [ARROW-1325|https://issues.apache.org/jira/browse/ARROW-1325], the > project contains a minimal working R package that takes advantage of R's > built-in support for calling C++ code and uses Rcpp for added support. > Though the exact public interface of the package hasn't been developed yet > (so this issue may be running before we walk), one major feature in its > development will be release to [CRAN|https://cran.r-project.org/], the > official package repository for the R language. > Completing this story would mean: > * Getting source code to state where [R CMD > CHECK|https://www.r-bloggers.com/how-i-learned-to-stop-worrying-and-love-r-cmd-check/] > passes with 0 ERRORS, 0 WARNINGS, and 0 NOTES > * Setting up build process for source and precompiled binary packages (see > [data.table|https://github.com/Rdatatable/data.table] and > [XGBoost|https://github.com/dmlc/xgboost] as examples of packages that do > this) > * Submission to CRAN and acceptance of the first release > > Distribution via CRAN would be a much more natural fit for most R users' > workflows than the current build-from-source workflow, and comes with all the > other benefits of package managers (e.g. version pegging, easy distribution > of platform-specific binaries). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-3759) [R] Build and test on Windows in Appveyor
[ https://issues.apache.org/jira/browse/ARROW-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-3759: Summary: [R] Build and test on Windows in Appveyor (was: [R] Run test suite on Windows in Appveyor) > [R] Build and test on Windows in Appveyor > - > > Key: ARROW-3759 > URL: https://issues.apache.org/jira/browse/ARROW-3759 > Project: Apache Arrow > Issue Type: Improvement > Components: R >Reporter: Wes McKinney >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-3351) [Python] Build failure on macOS
[
https://issues.apache.org/jira/browse/ARROW-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-3351.
---
> [Python] Build failure on macOS
> ---
>
> Key: ARROW-3351
> URL: https://issues.apache.org/jira/browse/ARROW-3351
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Wes McKinney
>Priority: Major
>
> Just started seeing this in Travis CI
> {code}
> CMake Error at cmake_modules/FindNumPy.cmake:62 (message):
> NumPy import failure:
> dyld: lazy symbol binding failed: Symbol not found: _getentropy
> Referenced from:
> /Users/travis/build/apache/arrow/pyarrow-test-2.7/lib/libpython2.7.dylib
> Expected in: /usr/lib/libSystem.B.dylib
>
> dyld: Symbol not found: _getentropy
> Referenced from:
> /Users/travis/build/apache/arrow/pyarrow-test-2.7/lib/libpython2.7.dylib
> Expected in: /usr/lib/libSystem.B.dylib
>
> Call Stack (most recent call first):
> src/arrow/python/CMakeLists.txt:23 (find_package)
> -- Configuring incomplete, errors occurred!
> See also
> "/Users/travis/build/apache/arrow/cpp-build/CMakeFiles/CMakeOutput.log".
> See also
> "/Users/travis/build/apache/arrow/cpp-build/CMakeFiles/CMakeError.log".
> The command "$TRAVIS_BUILD_DIR/ci/travis_script_python.sh 2.7" exited with 1.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
