[arrow] branch master updated: ARROW-3758: [R] Build R library and dependencies on Windows in Appveyor CI
This is an automated email from the ASF dual-hosted git repository. wesm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new 1d588ab ARROW-3758: [R] Build R library and dependencies on Windows in Appveyor CI 1d588ab is described below commit 1d588ab35241792f2e702e90ba331e9dc659354a Author: Neal Richardson AuthorDate: Fri Jun 21 19:34:43 2019 -0500 ARROW-3758: [R] Build R library and dependencies on Windows in Appveyor CI After much trial and error, I have successfully built the C++ library using the R toolchain on our Appveyor, and then used that in installing and checking the R package. Here is a passing build: https://ci.appveyor.com/project/nealrichardson/arrow/builds/25406324 Importantly, this patch puts the Windows Rtools/mingw pkgbuild script under our source control and continuous integration, as well as the additional complex procedure for assembling the "rwinlib" build described [on the wiki](https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages). In the process, this patch also updates the `r/configure.win` script to work with the latest version of Arrow and it addresses some test failures [...] One side effect of this CI setup is that the binary build artifacts (e.g. https://ci.appveyor.com/api/buildjobs/rkp1pu42dr2d5yev/artifacts/build%2Farrow-0.13.0.9000.zip) are now available for download and use on every commit. It is a small step from here to publishing those dev artifacts on each commit (as Jeroen's rtools-packages/backports repos to) or doing so nightly. This would enable R users/developers on Windows to install the development version of `arrow` without requiring the [...] Items resolved since the initial PR: - [x] Need to work out the right Windows incantations to find and pass in the path to the built C++ library to the R build script. The way I'm doing it now isn't totally kosher, but it's good enough for CI purposes and wouldn't affect the CRAN procedure. This is possibly deferable. - [x] Cleanup: need to uncomment the other appveyor jobs, obviously - [x] Add Windows dev installation instructions to the R README, currently nonexistent. (Doing on https://issues.apache.org/jira/browse/ARROW-) - [x] I need to confirm (with Jeroen) that it's ok to put the Apache license on the PKGBUILD script (copied with some modifications from [rtools-backports](https://github.com/r-windows/rtools-backports/blob/master/mingw-w64-arrow/PKGBUILD)) and appveyor-build-r.sh, which [is inspired by rtools-backports](https://github.com/r-windows/rtools-backports/blob/master/ci-build.sh) too. Seems uncontroversial to me but IANAL. Followup items: * The C++ library was built without Snappy support, which means that Parquet support will be limited in practice. I had to skip the one test that used a fixture .parquet file (which happened to use snappy compression). https://issues.apache.org/jira/browse/ARROW-5683 to address that. * I'd like to review this with Jeroen, the RTools maintainer. This works in the Appveyor job, and I've installed the R package on my local VM using the build that the Appveyor job creates, but there are a lot of subtleties in the R Windows toolchain and I am not confident that I've gotten them all right. https://issues.apache.org/jira/browse/ARROW-5686 Author: Neal Richardson Closes #4622 from nealrichardson/r-appveyor-part2 and squashes the following commits: 294a9797f Review feedback 9009969d9 Merge remote-tracking branch 'upstream/master' into r-appveyor-part2 234dc93fb Merge remote-tracking branch 'upstream/master' into r-appveyor-part2 c39d7e9c7 Some cleanup 9f560f1d2 Try absolute path; add more debug messages f5f6da2f9 Try this to pass in the local rwinlib 14520fdfc Read version from DESCRIPTION in build script; look elsewhere to set env var 51a6069b1 Merge upstream/master 976bdb291 Skip tests that fail on windows. Attempt to pass in local built rwinlib via env var cce770092 Oops 1ba73c2c4 Fix wonky URLs b302acfa3 Fix path, store .zip artifact e4e127bff Assemble built libraries for rwinlib. Adjust configure.win flags. Temporary hack to test out the R package with the built libs 21304b9c5 quote c7cdfd580 Try to collect the built packages as artifacts 46cf050a5 Try installing Antoine's patch a2800109c Now that that passes, update PKGBUILD to build from master 9c44946c4 Reset PKGBUILD to backports version 994c394cf this is the dir 3f0d2441e prints 46181a0fd more sigh 27aaf7fef try to find dir again 5de658bdf Sigh 72372a658 See if building 0.13 works 541d4555b ? c2eacc759 '' ed554f487 Try adding PATH 983345023
[arrow] branch master updated: ARROW-5537: [JS] Support delta dictionaries in RecordBatchWriter and DictionaryBuilder
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 40fcebe ARROW-5537: [JS] Support delta dictionaries in
RecordBatchWriter and DictionaryBuilder
40fcebe is described below
commit 40fcebea19140b4275757bd4b94f49918a0ce365
Author: ptaylor
AuthorDate: Fri Jun 21 17:26:41 2019 -0700
ARROW-5537: [JS] Support delta dictionaries in RecordBatchWriter and
DictionaryBuilder
Adds support for building and writing delta dictionaries. Moves the
`dictionary` Vector pointer to the Data class, similar to
https://github.com/apache/arrow/pull/4316.
Forked from https://github.com/apache/arrow/pull/4476 since this adds
support for delta dictionaries to the DictionaryBuilder. Will rebase this PR
after that's merged. All the work is in the last commit, here:
https://github.com/apache/arrow/commit/b12d842ebe1b341b2b0c67525bcf3f85ea88fecd
Author: ptaylor
Closes #4502 from trxcllnt/js/delta-dictionaries and squashes the following
commits:
6a70a25bd make dictionarybuilder and recordbatchwriter support
delta dictionaries
---
js/bin/print-buffer-alignment.js | 4 +-
js/src/builder.ts | 16 +---
js/src/builder/dictionary.ts | 36 ++---
js/src/column.ts | 2 +-
js/src/data.ts | 19 +++--
js/src/interfaces.ts | 4 +-
js/src/io/node/builder.ts | 15
js/src/io/whatwg/builder.ts| 16
js/src/ipc/metadata/json.ts| 29 ---
js/src/ipc/metadata/message.ts | 29 ---
js/src/ipc/reader.ts | 16 ++--
js/src/ipc/writer.ts | 102 +++--
js/src/recordbatch.ts | 33 +++-
js/src/schema.ts | 37 +++--
js/src/type.ts | 6 +-
js/src/util/args.ts| 6 +-
js/src/vector/chunked.ts | 6 +-
js/src/vector/date.ts | 43 +--
js/src/vector/dictionary.ts| 12 +--
js/src/visitor/vectorloader.ts | 14 +++-
js/test/data/tables.ts | 1 -
js/test/generate-test-data.ts | 25 +++---
js/test/unit/ipc/message-reader-tests.ts | 19 ++---
js/test/unit/ipc/writer/stream-writer-tests.ts | 45 ++-
js/test/unit/table-tests.ts| 24 +++---
js/test/unit/vector/vector-tests.ts| 4 +-
26 files changed, 300 insertions(+), 263 deletions(-)
diff --git a/js/bin/print-buffer-alignment.js b/js/bin/print-buffer-alignment.js
index fe7f493..4c32603 100755
--- a/js/bin/print-buffer-alignment.js
+++ b/js/bin/print-buffer-alignment.js
@@ -73,9 +73,9 @@ const { VectorLoader } =
require(`../targets/apache-arrow/visitor/vectorloader`)
})().catch((e) => { console.error(e); process.exit(1); });
function loadRecordBatch(schema, header, body) {
-return new RecordBatch(schema, header.length, new VectorLoader(body,
header.nodes, header.buffers).visitMany(schema.fields));
+return new RecordBatch(schema, header.length, new VectorLoader(body,
header.nodes, header.buffers, new Map()).visitMany(schema.fields));
}
function loadDictionaryBatch(header, body, dictionaryType) {
-return RecordBatch.new(new VectorLoader(body, header.nodes,
header.buffers).visitMany([dictionaryType]));
+return RecordBatch.new(new VectorLoader(body, header.nodes,
header.buffers, new Map()).visitMany([dictionaryType]));
}
diff --git a/js/src/builder.ts b/js/src/builder.ts
index 6f12afc..4251ce4 100644
--- a/js/src/builder.ts
+++ b/js/src/builder.ts
@@ -157,13 +157,7 @@ export abstract class Builder {
* @nocollapse
*/
public static throughIterable(options: IterableBuilderOptions) {
-const build = throughIterable(options);
-if (!DataType.isDictionary(options.type)) {
-return build;
-}
-return function*(source: Iterable) {
-const chunks = []; for (const chunk of build(source)) {
chunks.push(chunk); } yield* chunks;
-};
+return throughIterable(options);
}
/**
@@ -192,13 +186,7 @@ export abstract class Builder {
* @nocollapse
*/
public static throughAsyncIterable(options: IterableBuilderOptions) {
-const build = throughAsyncIterable(options);
-if (!DataType.isDictionary(options.type)) {
-return build;
-}
-return async function* (source: Iterable |
AsyncIterable) {
-const chunks = []; for await (const chunk of build(source)) {
[arrow] branch master updated: ARROW-5438: [JS] EOS bytes for sequential readers
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new 71deb04 ARROW-5438: [JS] EOS bytes for sequential readers 71deb04 is described below commit 71deb04536b8fdc98a8f99433555af2ac1b81072 Author: John Muehlhausen AuthorDate: Fri Jun 21 17:06:22 2019 -0700 ARROW-5438: [JS] EOS bytes for sequential readers Author: John Muehlhausen Closes #4466 from jgm-ktg/arrow_5438 and squashes the following commits: d8a73e370 ARROW-5438: EOS bytes for sequential readers --- js/src/ipc/writer.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/js/src/ipc/writer.ts b/js/src/ipc/writer.ts index 363d588..14a8d0f 100644 --- a/js/src/ipc/writer.ts +++ b/js/src/ipc/writer.ts @@ -324,6 +324,7 @@ export class RecordBatchFileWriter this._recordBatchBlocks, this._dictionaryBlocks )); return this +._writePadding(4) // EOS bytes for sequential readers ._write(buffer) // Write the flatbuffer ._write(Int32Array.of(buffer.byteLength)) // then the footer size suffix ._writeMagic(); // then the magic suffix
[arrow] branch master updated: ARROW-5631: [C++] Fix FindBoost targets with cmake3.2
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 00db153 ARROW-5631: [C++] Fix FindBoost targets with cmake3.2
00db153 is described below
commit 00db153824475800a2a0974a5180d669b156b3ec
Author: François Saint-Jacques
AuthorDate: Fri Jun 21 17:17:51 2019 -0500
ARROW-5631: [C++] Fix FindBoost targets with cmake3.2
cmake's FindBoost pre 3.5 did not export IMPORTED targets. This also fixes
the newly introduced trusty docker image.
Author: François Saint-Jacques
Author: Wes McKinney
Closes #4605 from fsaintjacques/ARROW-5631-cmake32-boost and squashes the
following commits:
9b946dad7 Code review feedback
01f67dd65 Fix R docker image
6c632072d Fix c_glib PKG_CONFIG_PATH in docker
3b70ff305 Fix rust docker image
4b1c29fd8 ARROW-5653: Fix conda libraries
7544850a4 Remove debug docker command
bd85741cc Add custom docker and vendor boost
78e41cc4b Fix trusty docker build
---
ci/docker_build_and_test_cpp.sh | 6
cpp/Dockerfile.cmake| 68 +
cpp/Dockerfile.ubuntu-trusty| 1 +
dev/tasks/tests.yml | 2 +-
docker-compose.yml | 8 +++--
r/Dockerfile| 18 ++-
r/tests/testthat/test-json.R| 4 +--
rust/Dockerfile | 3 ++
8 files changed, 98 insertions(+), 12 deletions(-)
diff --git a/ci/docker_build_and_test_cpp.sh b/ci/docker_build_and_test_cpp.sh
index 99b9460..f2b9f54 100755
--- a/ci/docker_build_and_test_cpp.sh
+++ b/ci/docker_build_and_test_cpp.sh
@@ -24,5 +24,11 @@ pushd /build/cpp
export ARROW_TEST_DATA=/arrow/testing/data
export PARQUET_TEST_DATA=/arrow/cpp/submodules/parquet-testing/data
+# ARROW-5653 after install target, RPATH is modified on linux. Add this such
+# that libraries are found with conda.
+if [[ ! -z "${CONDA_PREFIX}" ]]; then
+ export LD_LIBRARY_PATH=${CONDA_PREFIX}/lib
+fi
+
ninja unittest
popd
diff --git a/cpp/Dockerfile.cmake b/cpp/Dockerfile.cmake
new file mode 100644
index 000..9160cd7
--- /dev/null
+++ b/cpp/Dockerfile.cmake
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM ubuntu:18.04
+
+# install build essentials
+RUN export DEBIAN_FRONTEND=noninteractive && \
+apt-get update -y -q && \
+apt-get install -y -q --no-install-recommends \
+ca-certificates \
+curl \
+ccache \
+g++ \
+gcc \
+git \
+libidn11 \
+ninja-build \
+pkg-config \
+tzdata \
+wget \
+&& apt-get clean \
+&& rm -rf /var/lib/apt/lists/*
+
+# install conda and required packages
+ARG EXTRA_CONDA_PKGS
+ENV PATH=/opt/conda/bin:$PATH \
+CONDA_PREFIX=/opt/conda
+COPY ci/docker_install_conda.sh \
+ ci/conda_env_cpp.yml \
+ ci/conda_env_unix.yml \
+ /arrow/ci/
+RUN arrow/ci/docker_install_conda.sh && \
+conda install -q -c conda-forge \
+--file arrow/ci/conda_env_cpp.yml \
+--file arrow/ci/conda_env_unix.yml \
+$EXTRA_CONDA_PKGS && \
+conda clean --all
+
+ARG CMAKE_VERSION=3.2.3
+ARG
CMAKE_URL=https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-Linux-x86_64.tar.gz
+RUN curl ${CMAKE_URL} -Lo cmake-${CMAKE_VERSION}.tar.gz && \
+ mkdir /opt/cmake && tar -zxvf cmake-${CMAKE_VERSION}.tar.gz -C /opt/cmake
+ENV PATH=/opt/cmake/cmake-${CMAKE_VERSION}-Linux-x86_64/bin:$PATH
+
+ENV CC=gcc \
+CXX=g++ \
+ARROW_GANDIVA=OFF \
+ARROW_BUILD_TESTS=ON \
+ARROW_DEPENDENCY_SOURCE=CONDA \
+ARROW_HOME=$CONDA_PREFIX \
+PARQUET_HOME=$CONDA_PREFIX
+
+# build and test
+CMD ["arrow/ci/docker_build_and_test_cpp.sh"]
diff --git a/cpp/Dockerfile.ubuntu-trusty b/cpp/Dockerfile.ubuntu-trusty
index d44491c..bca5177 100644
--- a/cpp/Dockerfile.ubuntu-trusty
+++ b/cpp/Dockerfile.ubuntu-trusty
@@ -62,6 +62,7 @@ ENV CC=gcc \
ARROW_BUILD_TESTS=ON \
ARROW_DEPENDENCY_SOURCE=SYSTEM \
ARROW_FLIGHT=OFF \
+ ARROW_WITH_OPENSSL=OFF \
[arrow] branch master updated: ARROW-3676: [Go] implement Decimal128 array
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new f406721 ARROW-3676: [Go] implement Decimal128 array
f406721 is described below
commit f40672143cf9d0989083a654a33b7b7c0725ed75
Author: Sebastien Binet
AuthorDate: Fri Jun 21 16:59:41 2019 -0500
ARROW-3676: [Go] implement Decimal128 array
Author: Sebastien Binet
Closes #4633 from sbinet/issue-3676 and squashes the following commits:
eb62bc712 go/arrow/array: exercize more of Decimal128 API
surface
0ea83919b go/arrow/array: use Decimal128.Value in Stringer
9b57f7425 go/arrow/decimal128: test hi/lo bits
7778a48a6 ARROW-3676: implement Decimal128 array
---
go/arrow/array/array.go| 2 +-
go/arrow/array/array_test.go | 11 +-
go/arrow/array/compare.go | 6 +
go/arrow/array/decimal128.go | 235 +
go/arrow/array/decimal128_test.go | 179 +
go/arrow/datatype_fixedwidth.go| 18 ++-
go/arrow/datatype_fixedwidth_test.go | 27
go/arrow/decimal128/decimal128.go | 73 ++
go/arrow/decimal128/decimal128_test.go | 94 +
go/arrow/type_traits_decimal128.go | 75 +++
go/arrow/type_traits_test.go | 45 +++
11 files changed, 759 insertions(+), 6 deletions(-)
diff --git a/go/arrow/array/array.go b/go/arrow/array/array.go
index 1912f3e..d8418f8 100644
--- a/go/arrow/array/array.go
+++ b/go/arrow/array/array.go
@@ -186,7 +186,7 @@ func init() {
arrow.TIME32:func(data *Data) Interface { return
NewTime32Data(data) },
arrow.TIME64:func(data *Data) Interface { return
NewTime64Data(data) },
arrow.INTERVAL: func(data *Data) Interface { return
NewIntervalData(data) },
- arrow.DECIMAL: unsupportedArrayType,
+ arrow.DECIMAL: func(data *Data) Interface { return
NewDecimal128Data(data) },
arrow.LIST: func(data *Data) Interface { return
NewListData(data) },
arrow.STRUCT:func(data *Data) Interface { return
NewStructData(data) },
arrow.UNION: unsupportedArrayType,
diff --git a/go/arrow/array/array_test.go b/go/arrow/array/array_test.go
index 724f3b4..ba3a961 100644
--- a/go/arrow/array/array_test.go
+++ b/go/arrow/array/array_test.go
@@ -43,9 +43,6 @@ func TestMakeFromData(t *testing.T) {
expPanic bool
expError string
}{
- // unsupported types
- {name: "map", d: {arrow.MAP}, expPanic: true,
expError: "unsupported data type: MAP"},
-
// supported types
{name: "null", d: {arrow.NULL}},
{name: "bool", d: {arrow.BOOL}},
@@ -59,11 +56,17 @@ func TestMakeFromData(t *testing.T) {
{name: "int64", d: {arrow.INT64}},
{name: "float32", d: {arrow.FLOAT32}},
{name: "float64", d: {arrow.FLOAT64}},
+ {name: "string", d: {arrow.STRING}, size: 3},
{name: "binary", d: {arrow.BINARY}, size: 3},
+ {name: "fixed_size_binary", d:
{arrow.FIXED_SIZE_BINARY}},
+ {name: "date32", d: {arrow.DATE32}},
+ {name: "date64", d: {arrow.DATE64}},
{name: "timestamp", d: {arrow.TIMESTAMP}},
{name: "time32", d: {arrow.TIME32}},
{name: "time64", d: {arrow.TIME64}},
- {name: "fixed_size_binary", d:
{arrow.FIXED_SIZE_BINARY}},
+ {name: "month_interval", d:
arrow.FixedWidthTypes.MonthInterval},
+ {name: "day_time_interval", d:
arrow.FixedWidthTypes.DayTimeInterval},
+ {name: "decimal", d: {arrow.DECIMAL}},
{name: "list", d: {arrow.LIST}, child:
[]*array.Data{
array.NewData({arrow.INT64}, 0,
make([]*memory.Buffer, 4), nil, 0, 0),
diff --git a/go/arrow/array/compare.go b/go/arrow/array/compare.go
index c6665c9..17839c6 100644
--- a/go/arrow/array/compare.go
+++ b/go/arrow/array/compare.go
@@ -128,6 +128,9 @@ func ArrayEqual(left, right Interface) bool {
case *Float64:
r := right.(*Float64)
return arrayEqualFloat64(l, r)
+ case *Decimal128:
+ r := right.(*Decimal128)
+ return arrayEqualDecimal128(l, r)
case *Date32:
r := right.(*Date32)
return arrayEqualDate32(l, r)
@@ -314,6 +317,9 @@ func arrayApproxEqual(left, right Interface, opt
equalOption) bool {
case *Float64:
r := right.(*Float64)
return arrayApproxEqualFloat64(l, r, opt)
+ case *Decimal128:
+
[arrow] branch master updated: ARROW-5605: [C++] Verify Flatbuffer messages in more places to prevent crashes due to bad inputs
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new b04fae1 ARROW-5605: [C++] Verify Flatbuffer messages in more places
to prevent crashes due to bad inputs
b04fae1 is described below
commit b04fae1cb65e56637cb28c6229c65a356e0de147
Author: Marco Neumann
AuthorDate: Fri Jun 21 16:59:06 2019 -0500
ARROW-5605: [C++] Verify Flatbuffer messages in more places to prevent
crashes due to bad inputs
While the first commit (`fix ReadRecordBatch validation`) is sufficient to
fix ARROW-5605, I took the time to fix very similar issues in the code IPC code
base, so our users are probably protected and also to help the fuzzer to not
run straight into a similar problem again.
Author: Marco Neumann
Author: Marco Neumann
Author: Wes McKinney
Closes #4573 from crepererum/ARROW-5605 and squashes the following commits:
75a3b6d2f Refactor message verification into helper function
6b528d48b fix linting
80f6ce2a4 remove risky reinterpret_casts
7ebdd7559 validate footer
fb70669b1 remove obsolete reinterpret_cast
cdb79849c guard all getMessage calls with validation
27d7dad46 fix ReadRecordBatch validation
---
cpp/src/arrow/ipc/message.cc | 20 +++-
cpp/src/arrow/ipc/metadata-internal.cc | 22 +
cpp/src/arrow/ipc/metadata-internal.h | 11 +
cpp/src/arrow/ipc/reader.cc| 43 +-
4 files changed, 58 insertions(+), 38 deletions(-)
diff --git a/cpp/src/arrow/ipc/message.cc b/cpp/src/arrow/ipc/message.cc
index f6d43e1..1de2a26 100644
--- a/cpp/src/arrow/ipc/message.cc
+++ b/cpp/src/arrow/ipc/message.cc
@@ -43,7 +43,8 @@ class Message::MessageImpl {
: metadata_(metadata), message_(nullptr), body_(body) {}
Status Open() {
-message_ = flatbuf::GetMessage(metadata_->data());
+RETURN_NOT_OK(
+internal::VerifyMessage(metadata_->data(), metadata_->size(),
_));
// Check that the metadata version is supported
if (message_->version() < internal::kMinMetadataVersion) {
@@ -143,12 +144,8 @@ bool Message::Equals(const Message& other) const {
Status Message::ReadFrom(const std::shared_ptr& metadata,
io::InputStream* stream,
std::unique_ptr* out) {
- auto data = metadata->data();
- flatbuffers::Verifier verifier(data, metadata->size(), /*max_depth=*/128);
- if (!flatbuf::VerifyMessageBuffer(verifier)) {
-return Status::IOError("Invalid flatbuffers message.");
- }
- auto fb_message = flatbuf::GetMessage(data);
+ const flatbuf::Message* fb_message;
+ RETURN_NOT_OK(internal::VerifyMessage(metadata->data(), metadata->size(),
_message));
int64_t body_length = fb_message->bodyLength();
@@ -164,8 +161,8 @@ Status Message::ReadFrom(const std::shared_ptr&
metadata, io::InputStrea
Status Message::ReadFrom(const int64_t offset, const std::shared_ptr&
metadata,
io::RandomAccessFile* file, std::unique_ptr*
out) {
- auto fb_message = flatbuf::GetMessage(metadata->data());
-
+ const flatbuf::Message* fb_message;
+ RETURN_NOT_OK(internal::VerifyMessage(metadata->data(), metadata->size(),
_message));
int64_t body_length = fb_message->bodyLength();
std::shared_ptr body;
@@ -209,9 +206,8 @@ Status Message::SerializeTo(io::OutputStream* stream,
int32_t alignment,
}
bool Message::Verify() const {
- std::shared_ptr meta = this->metadata();
- flatbuffers::Verifier verifier(meta->data(), meta->size(), 128);
- return flatbuf::VerifyMessageBuffer(verifier);
+ const flatbuf::Message* unused;
+ return internal::VerifyMessage(metadata()->data(), metadata()->size(),
).ok();
}
std::string FormatMessageType(Message::Type type) {
diff --git a/cpp/src/arrow/ipc/metadata-internal.cc
b/cpp/src/arrow/ipc/metadata-internal.cc
index 46f3366..4b349cb 100644
--- a/cpp/src/arrow/ipc/metadata-internal.cc
+++ b/cpp/src/arrow/ipc/metadata-internal.cc
@@ -1112,8 +1112,12 @@ Status GetSchema(const void* opaque_schema,
DictionaryMemo* dictionary_memo,
Status GetTensorMetadata(const Buffer& metadata, std::shared_ptr*
type,
std::vector* shape, std::vector*
strides,
std::vector* dim_names) {
- auto message = flatbuf::GetMessage(metadata.data());
- auto tensor = reinterpret_cast(message->header());
+ const flatbuf::Message* message;
+ RETURN_NOT_OK(internal::VerifyMessage(metadata.data(), metadata.size(),
));
+ auto tensor = message->header_as_Tensor();
+ if (tensor == nullptr) {
+return Status::IOError("Header-type of flatbuffer-encoded Message is not
Tensor.");
+ }
int ndim = static_cast(tensor->shape()->size());
@@ -1143,15 +1147,13 @@ Status GetSparseTensorMetadata(const Buffer& metadata,
std::shared_ptr
[arrow] branch master updated: ARROW-5678: [R][Lint] Fix hadolint docker linting error
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 9fd7357 ARROW-5678: [R][Lint] Fix hadolint docker linting error
9fd7357 is described below
commit 9fd7357757bc1dcd1e1c8e4612a2f1bea38ccd04
Author: Wes McKinney
AuthorDate: Fri Jun 21 12:44:59 2019 -0500
ARROW-5678: [R][Lint] Fix hadolint docker linting error
Author: Wes McKinney
Author: Krisztián Szűcs
Closes #4650 from kszucs/r-hadolint and squashes the following commits:
be8e587dc Ignore ruby gem version pinning
1558bc61e fix docker linting error
---
.hadolint.yaml | 2 ++
r/Dockerfile | 26 ++
2 files changed, 16 insertions(+), 12 deletions(-)
diff --git a/.hadolint.yaml b/.hadolint.yaml
index 33190e8..99ce671 100644
--- a/.hadolint.yaml
+++ b/.hadolint.yaml
@@ -19,3 +19,5 @@ ignored:
- DL3008
- DL3013
- DL3018
+ - DL3015 # Avoid additional packages by specifying `--no-install-recommends`
+ - DL3028 # Ruby gem version pinning
diff --git a/r/Dockerfile b/r/Dockerfile
index 7c0b087..63ff3e7 100644
--- a/r/Dockerfile
+++ b/r/Dockerfile
@@ -27,27 +27,29 @@ ENV DEBIAN_FRONTEND=noninteractive
# Build R
# [1]
https://www.digitalocean.com/community/tutorials/how-to-install-r-on-ubuntu-18-04
# [2]
https://linuxize.com/post/how-to-install-r-on-ubuntu-18-04/#installing-r-packages-from-cran
-RUN apt update && \
-apt install -y \
+RUN apt-get update -y && \
+apt-get install -y \
apt-transport-https \
software-properties-common && \
apt-key adv \
--keyserver keyserver.ubuntu.com \
--recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu
bionic-cran35/' && \
-apt install -y r-base && \
+apt-get install -y r-base && \
# system libs needed by core R packages
-apt install -y \
-libgit2-dev \
-libssl-dev && \
+apt-get install -y \
+libgit2-dev \
+libssl-dev && \
# install clang to mirror what was done on Travis
-apt install -y \
-clang \
-clang-format \
-clang-tidy && \
+apt-get install -y \
+clang \
+clang-format \
+clang-tidy && \
# R CMD CHECK --as-cran needs pdflatex to build the package manual
-apt install -y \
-texlive-latex-base && \
+apt-get install -y \
+texlive-latex-base && \
+apt-get clean && \
+rm -rf /var/lib/apt/lists/* && \
Rscript -e "install.packages('devtools', repos =
'http://cran.rstudio.com')" && \
Rscript -e "devtools::install_github('romainfrancois/decor')" && \
Rscript -e "install.packages(c( \
[arrow] branch master updated: ARROW-5668 [C++/Python] Include 'not null' in schema fields pretty print
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new a6d1b9c ARROW-5668 [C++/Python] Include 'not null' in schema fields
pretty print
a6d1b9c is described below
commit a6d1b9c789af3108c95263c97cea5d9fd3cde760
Author: Joris Van den Bossche
AuthorDate: Fri Jun 21 12:09:07 2019 -0500
ARROW-5668 [C++/Python] Include 'not null' in schema fields pretty print
https://issues.apache.org/jira/browse/ARROW-5668
Author: Joris Van den Bossche
Closes #4645 from jorisvandenbossche/ARROW-5668-schema-repr-nonnull and
squashes the following commits:
055917f98 lint
bfc6c6842 ARROW-5668 include 'not null' in schema
fields repr
---
cpp/src/arrow/pretty_print-test.cc | 23 +++
cpp/src/arrow/pretty_print.cc | 9 ++---
2 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/cpp/src/arrow/pretty_print-test.cc
b/cpp/src/arrow/pretty_print-test.cc
index 1aed4d7..7600ab4 100644
--- a/cpp/src/arrow/pretty_print-test.cc
+++ b/cpp/src/arrow/pretty_print-test.cc
@@ -658,4 +658,27 @@ four: struct
+ child 0, item: int32
+four: list not null
+ child 0, item: int32
+five: list
+ child 0, item: int32 not null)expected";
+
+ PrettyPrintOptions options{0};
+
+ Check(*sch, options, expected);
+}
+
} // namespace arrow
diff --git a/cpp/src/arrow/pretty_print.cc b/cpp/src/arrow/pretty_print.cc
index 175514b..cb67b0d 100644
--- a/cpp/src/arrow/pretty_print.cc
+++ b/cpp/src/arrow/pretty_print.cc
@@ -570,7 +570,7 @@ class SchemaPrinter : public PrettyPrinter {
: PrettyPrinter(indent, indent_size, window, skip_new_lines, sink),
schema_(schema) {}
- Status PrintType(const DataType& type);
+ Status PrintType(const DataType& type, bool nullable);
Status PrintField(const Field& field);
Status Print() {
@@ -588,8 +588,11 @@ class SchemaPrinter : public PrettyPrinter {
const Schema& schema_;
};
-Status SchemaPrinter::PrintType(const DataType& type) {
+Status SchemaPrinter::PrintType(const DataType& type, bool nullable) {
Write(type.ToString());
+ if (!nullable) {
+Write(" not null");
+ }
for (int i = 0; i < type.num_children(); ++i) {
Newline();
@@ -607,7 +610,7 @@ Status SchemaPrinter::PrintType(const DataType& type) {
Status SchemaPrinter::PrintField(const Field& field) {
Write(field.name());
Write(": ");
- return PrintType(*field.type());
+ return PrintType(*field.type(), field.nullable());
}
Status PrettyPrint(const Schema& schema, const PrettyPrintOptions& options,
[arrow] branch master updated: ARROW-5674: [Python] Missing pandas pytest markers from test_parquet.py
This is an automated email from the ASF dual-hosted git repository. wesm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new be16de0 ARROW-5674: [Python] Missing pandas pytest markers from test_parquet.py be16de0 is described below commit be16de0bdcbf872412ab6d3da99514218ed3ce8b Author: Krisztián Szűcs AuthorDate: Fri Jun 21 12:07:35 2019 -0500 ARROW-5674: [Python] Missing pandas pytest markers from test_parquet.py Author: Krisztián Szűcs Closes #4646 from kszucs/ARROW-5674 and squashes the following commits: 023cee31e add pandas markers to test_parquet.py --- python/pyarrow/tests/test_parquet.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/python/pyarrow/tests/test_parquet.py b/python/pyarrow/tests/test_parquet.py index 49a03d6..9db04f7 100644 --- a/python/pyarrow/tests/test_parquet.py +++ b/python/pyarrow/tests/test_parquet.py @@ -2863,6 +2863,7 @@ def test_parquet_file_too_small(tempdir): pq.read_table(path) +@pytest.mark.pandas def test_multi_dataset_metadata(tempdir): filenames = ["ARROW-1983-dataset.0", "ARROW-1983-dataset.1"] metapath = str(tempdir / "_metadata") @@ -2905,6 +2906,7 @@ def test_multi_dataset_metadata(tempdir): assert md['serialized_size'] > 0 +@pytest.mark.pandas def test_filter_before_validate_schema(tempdir): # ARROW-4076 apply filter before schema validation # to avoid checking unneeded schemas
[arrow] branch master updated: ARROW-5656: [Python][Packaging] Fix macOS wheel builds, add Flight support
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 3a37bf2 ARROW-5656: [Python][Packaging] Fix macOS wheel builds, add
Flight support
3a37bf2 is described below
commit 3a37bf29c512b4c72c8da5b2a8657b21548cc47a
Author: Wes McKinney
AuthorDate: Fri Jun 21 12:02:24 2019 -0500
ARROW-5656: [Python][Packaging] Fix macOS wheel builds, add Flight support
This fixes a regression introduced by the shared library duplication issue
that was fixed in Linux recently.
Author: Wes McKinney
Author: Philipp Moritz
Closes #4642 from wesm/ARROW-5656 and squashes the following commits:
71e277664 Review feedback, try to fix Python 3.5 windows
wheels
7123ed6cb Fix git submodule call
7e9d2c565 Don't add ABI version to Boost dylibs on macOS for
now
53dc41530 cmake-format
1fe05a9be Get gRPC using SYSTEM method, set ARROW_TEST_DATA
env variable
497ba3f33 Use gRPC from Homebrew
4f583e8f5 install requests package
a4db1ac6b Use requests in get_apache_mirror.py if available
db0caab28 First attempt to enable Flight in macOS wheels
---
cpp/Brewfile | 1 +
cpp/build-support/get_apache_mirror.py | 19 ++-
dev/tasks/python-wheels/osx-build.sh | 11 ++-
dev/tasks/python-wheels/travis.osx.yml | 9 -
dev/tasks/python-wheels/win-build.bat | 5 -
python/CMakeLists.txt | 2 +-
python/requirements-wheel.txt | 1 +
7 files changed, 39 insertions(+), 9 deletions(-)
diff --git a/cpp/Brewfile b/cpp/Brewfile
index c658887..4b796db 100644
--- a/cpp/Brewfile
+++ b/cpp/Brewfile
@@ -35,6 +35,7 @@ brew "python"
brew "rapidjson"
brew "re2"
brew "snappy"
+brew "openssl"
brew "thrift"
brew "wget"
brew "zstd"
diff --git a/cpp/build-support/get_apache_mirror.py
b/cpp/build-support/get_apache_mirror.py
index ac52255..38ea6f4 100755
--- a/cpp/build-support/get_apache_mirror.py
+++ b/cpp/build-support/get_apache_mirror.py
@@ -21,11 +21,20 @@
import json
try:
-from urllib2 import urlopen
+import requests
+
+def get_url(url):
+return requests.get(url).content
except ImportError:
-# py3
-from urllib.request import urlopen
+try:
+from urllib2 import urlopen
+except ImportError:
+# py3
+from urllib.request import urlopen
+
+def get_url(url):
+return urlopen(url).read()
-suggested_mirror = urlopen('https://www.apache.org/dyn/'
- 'closer.cgi?as_json=1').read()
+suggested_mirror = get_url('https://www.apache.org/dyn/'
+ 'closer.cgi?as_json=1')
print(json.loads(suggested_mirror.decode('utf-8'))['preferred'])
diff --git a/dev/tasks/python-wheels/osx-build.sh
b/dev/tasks/python-wheels/osx-build.sh
index 9fef5b9..be39754 100755
--- a/dev/tasks/python-wheels/osx-build.sh
+++ b/dev/tasks/python-wheels/osx-build.sh
@@ -22,7 +22,9 @@ set -e
# overrides multibuild's default build_wheel
function build_wheel {
pip install -U pip
-pip install setuptools_scm
+
+# ARROW-5670: Python 3.5 can fail with HTTPS error in CMake build
+pip install setuptools_scm requests
# Include brew installed versions of flex and bison.
# We need them to build Thrift. The ones that come with Xcode are too old.
@@ -111,6 +113,9 @@ function build_wheel {
export BUILD_ARROW_GANDIVA=OFF
fi
+git submodule update --init
+export ARROW_TEST_DATA=`pwd`/testing/data
+
pushd cpp
mkdir build
pushd build
@@ -129,6 +134,8 @@ function build_wheel {
-DARROW_ORC=ON \
-DBOOST_ROOT="$arrow_boost_dist" \
-DBoost_NAMESPACE=arrow_boost \
+ -DARROW_FLIGHT=ON \
+ -DgRPC_SOURCE=SYSTEM \
-DMAKE=make \
..
make -j5
@@ -143,6 +150,7 @@ function build_wheel {
unset ARROW_HOME
unset PARQUET_HOME
+export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_PLASMA=1
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_ORC=1
@@ -186,6 +194,7 @@ import pyarrow.parquet
import pyarrow.plasma
if sys.version_info.major > 2:
+import pyarrow.flight
import pyarrow.gandiva
"
diff --git a/dev/tasks/python-wheels/travis.osx.yml
b/dev/tasks/python-wheels/travis.osx.yml
index 919bcf5..2db3208 100644
--- a/dev/tasks/python-wheels/travis.osx.yml
+++ b/dev/tasks/python-wheels/travis.osx.yml
@@ -38,17 +38,24 @@ before_install:
- git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
- git -C arrow checkout {{ arrow.head }}
+ # ARROW-3976 Old versions of git can cause failures when Homebrew prints a
+ # donation solicitation. Attempt to update git
+ - git --version
+ - brew upgrade git
+
# Also remove artifacts that depend on Boost
- brew
[arrow] branch master updated (451445e -> 1718fe3)
This is an automated email from the ASF dual-hosted git repository. wesm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git. from 451445e ARROW-5675: [Doc] Fix typo in Xcode workflow documentation add 1718fe3 ARROW-5654: [C++][Python] Add ChunkedArray::Validate method that checks chunk types for consistency, invoke in Python No new revisions were added by this update. Summary of changes: cpp/src/arrow/table-test.cc | 21 + cpp/src/arrow/table.cc | 17 + cpp/src/arrow/table.h| 3 +++ python/pyarrow/includes/libarrow.pxd | 2 ++ python/pyarrow/table.pxi | 6 ++ python/pyarrow/tests/test_table.py | 7 ++- 6 files changed, 55 insertions(+), 1 deletion(-)
[arrow] branch master updated: ARROW-5675: [Doc] Fix typo in Xcode workflow documentation
This is an automated email from the ASF dual-hosted git repository. fsaintjacques pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new 451445e ARROW-5675: [Doc] Fix typo in Xcode workflow documentation 451445e is described below commit 451445e00f80fd126087b8b2ce90867f812f8438 Author: Hatem Helal AuthorDate: Fri Jun 21 09:37:33 2019 -0400 ARROW-5675: [Doc] Fix typo in Xcode workflow documentation This is a followup to address @kou's comment here: https://github.com/apache/arrow/pull/4596#discussion_r296093152 Author: Hatem Helal Closes #4647 from hatemhelal/arrow-5675 and squashes the following commits: ab05b8283 remove ^ --- docs/source/developers/cpp.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/source/developers/cpp.rst b/docs/source/developers/cpp.rst index 1244040..b44b0d6 100644 --- a/docs/source/developers/cpp.rst +++ b/docs/source/developers/cpp.rst @@ -649,8 +649,7 @@ by generating an Xcode project: cd cpp mkdir xcode-build cd xcode-build - cmake .. -G Xcode ^ - -DARROW_BUILD_TESTS=ON + cmake .. -G Xcode -DARROW_BUILD_TESTS=ON open arrow.xcodeproj This will generate a project and open it in the Xcode app. As an alternative,
[arrow] branch master updated: ARROW-5664: [Crossbow] Execute nightly crossbow tests on CircleCI instead of Travis
This is an automated email from the ASF dual-hosted git repository.
kszucs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new b9926b4 ARROW-5664: [Crossbow] Execute nightly crossbow tests on
CircleCI instead of Travis
b9926b4 is described below
commit b9926b4a7526e8b85758b976063669163371851d
Author: Krisztián Szűcs
AuthorDate: Fri Jun 21 11:26:32 2019 +0200
ARROW-5664: [Crossbow] Execute nightly crossbow tests on CircleCI instead
of Travis
The spark integration test has hit the 50mins maximum build time on travis,
whereas Circle can run jobs up to 5 hours. Now it's possible to run crossbow
tasks on circleci.
Author: Krisztián Szűcs
Closes #4631 from kszucs/crossbow-circleci and squashes the following
commits:
c5edebc25 update readme; change branch naming order; add
missing ci key to centos-7 build
d2b494454 unflatten within create_branch
0c0001063 run integration tests on circleci
---
dev/tasks/README.md | 10 ++-
dev/tasks/crossbow.py | 108 +---
dev/tasks/docker-tests/circle.linux.yml | 41
dev/tasks/tasks.yml | 28 +
dev/tasks/tests.yml | 75 ++
5 files changed, 214 insertions(+), 48 deletions(-)
diff --git a/dev/tasks/README.md b/dev/tasks/README.md
index 619d8a2..b68ae39 100644
--- a/dev/tasks/README.md
+++ b/dev/tasks/README.md
@@ -58,8 +58,10 @@ submission. The tasks are defined in `tasks.yml`
1. [Create the queue
repository](https://help.github.com/articles/creating-a-new-repository)
-2. Enable [TravisCI](https://travis-ci.org/getting_started) and
- [Appveyor](https://www.appveyor.com/docs/) integrations on it
+2. Enable [TravisCI](https://travis-ci.org/getting_started),
+ [Appveyor](https://www.appveyor.com/docs/) and
+ [CircleCI](https://circleci.com/docs/2.0/getting-started/)
+ integrations on it
- turn off Travis' [auto
cancellation](https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit)
feature on branches
@@ -92,6 +94,10 @@ submission. The tasks are defined in `tasks.yml`
- TravisCI: `https://travis-ci.org///settings`
- Appveyor:
`https://ci.appveyor.com/project///settings/environment`
+ - CircleCI: `https://circleci.com/gh///edit#env-vars`
+
+ On Appveyor check the `skip branches without appveyor.yml` checkbox on the
+ web UI under crossbow repository's settings.
7. Install Python 3.6:
diff --git a/dev/tasks/crossbow.py b/dev/tasks/crossbow.py
index e714188..b58ae9d 100755
--- a/dev/tasks/crossbow.py
+++ b/dev/tasks/crossbow.py
@@ -53,6 +53,62 @@ def md(template, *args, **kwargs):
return template.format(*map(escape, args), **toolz.valmap(escape, kwargs))
+def unflatten(mapping):
+result = {}
+for path, value in mapping.items():
+parents, leaf = path[:-1], path[-1]
+# create the hierarchy until we reach the leaf value
+temp = result
+for parent in parents:
+temp.setdefault(parent, {})
+temp = temp[parent]
+# set the leaf value
+temp[leaf] = value
+
+return result
+
+
+# configurations for setting up branch skipping
+# - appveyor has a feature to skip builds without an appveyor.yml
+# - travis reads from the master branch and applies the rules
+# - circle requires the configuration to be present on all branch, even ones
+# that are configured to be skipped
+
+_default_travis_yml = """
+branches:
+ only:
+- master
+- /.*-travis-.*/
+
+os: linux
+dist: trusty
+language: generic
+"""
+
+_default_circle_yml = """
+version: 2
+
+jobs:
+ build:
+machine: true
+
+workflows:
+ version: 2
+ build:
+jobs:
+ - build:
+ filters:
+branches:
+ only:
+- /.*-circle-.*/
+"""
+
+_default_tree = {
+'.travis.yml': _default_travis_yml,
+'.circleci/config.yml': _default_circle_yml
+}
+
+
class JiraChangelog:
def __init__(self, version, username, password,
@@ -263,17 +319,28 @@ class Repo:
return pygit2.Signature(self.user_name, self.user_email,
int(time.time()))
-def create_branch(self, branch_name, files, parents=[], message='',
- signature=None):
-# 1. create tree
+def create_tree(self, files):
builder = self.repo.TreeBuilder()
for filename, content in files.items():
-# insert the file and creating the new filetree
-blob_id = self.repo.create_blob(content)
-builder.insert(filename, blob_id, pygit2.GIT_FILEMODE_BLOB)
+if isinstance(content, dict):
+# create a subtree
+tree_id = self.create_tree(content)
+
[arrow] branch master updated: ARROW-5663: [Packaging][RPM] Update CentOS packages for 0.14.0
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new cffe335 ARROW-5663: [Packaging][RPM] Update CentOS packages for 0.14.0
cffe335 is described below
commit cffe33563ff59851d32ba7c0239c01b9bafa2c53
Author: Sutou Kouhei
AuthorDate: Fri Jun 21 15:22:28 2019 +0900
ARROW-5663: [Packaging][RPM] Update CentOS packages for 0.14.0
Author: Sutou Kouhei
Closes #4626 from kou/packaging-rpm and squashes the following commits:
79f0905ef Update CentOS packages for 0.14.0
---
cpp/cmake_modules/ThirdpartyToolchain.cmake | 8 ++-
dev/tasks/linux-packages/yum/arrow.spec.in | 83 +++-
dev/tasks/linux-packages/yum/centos-6/Dockerfile | 1 -
dev/tasks/linux-packages/yum/centos-7/Dockerfile | 9 +--
4 files changed, 64 insertions(+), 37 deletions(-)
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake
b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index e323668..df0839e 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -319,8 +319,10 @@ endif()
if(DEFINED ENV{ARROW_JEMALLOC_URL})
set(JEMALLOC_SOURCE_URL "$ENV{ARROW_JEMALLOC_URL}")
else()
- set(JEMALLOC_SOURCE_URL
-
"https://github.com/jemalloc/jemalloc/archive/${JEMALLOC_VERSION}.tar.gz;)
+ set(
+JEMALLOC_SOURCE_URL
+
"https://github.com/jemalloc/jemalloc/releases/download/${JEMALLOC_VERSION}/jemalloc-${JEMALLOC_VERSION}.tar.bz2;
+)
endif()
if(DEFINED ENV{ARROW_LZ4_URL})
@@ -1360,7 +1362,7 @@ if(ARROW_JEMALLOC)
jemalloc_ep
URL ${JEMALLOC_SOURCE_URL}
PATCH_COMMAND touch doc/jemalloc.3 doc/jemalloc.html
-CONFIGURE_COMMAND ./autogen.sh
+CONFIGURE_COMMAND ./configure
"AR=${CMAKE_AR}"
"CC=${CMAKE_C_COMPILER}"
"--prefix=${JEMALLOC_PREFIX}"
diff --git a/dev/tasks/linux-packages/yum/arrow.spec.in
b/dev/tasks/linux-packages/yum/arrow.spec.in
index d59aaed..452901f 100644
--- a/dev/tasks/linux-packages/yum/arrow.spec.in
+++ b/dev/tasks/linux-packages/yum/arrow.spec.in
@@ -19,7 +19,8 @@
%define _centos_ver %{?centos_ver:%{centos_ver}}%{!?centos_ver:7}
-%define use_python %{_centos_ver} >= 6
+%define boost_version 169
+%define python_version %(if [ "%{_centos_ver}" = 6 ]; then echo 34; else echo
36; fi)
%define use_parquet %{_centos_ver} >= 7
%define use_glib %{_centos_ver} >= 7
@@ -32,13 +33,16 @@ License:Apache-2.0
URL: https://arrow.apache.org/
Source0:
https://dist.apache.org/repos/dist/release/@PACKAGE@/@PACKAGE@-%{version}/apache-@PACKAGE@-%{version}.tar.gz
-BuildRequires: pkgconfig
-BuildRequires: gcc-c++
+BuildRequires: bison
+%if %{_centos_ver} >= 7
+BuildRequires: boost%{boost_version}-devel
+BuildRequires: brotli-devel
+%endif
BuildRequires: cmake3
-BuildRequires: boost-devel
%if %{_centos_ver} >= 7
BuildRequires: double-conversion-devel
%endif
+BuildRequires: gcc-c++
BuildRequires: gflags-devel
BuildRequires: git
%if %{_centos_ver} >= 7
@@ -46,20 +50,21 @@ BuildRequires: glog-devel
%endif
BuildRequires: libzstd-devel
BuildRequires: lz4-devel
+BuildRequires: pkgconfig
+BuildRequires: python%{python_version}-devel
+BuildRequires: python%{python_version}-numpy
%if %{_centos_ver} >= 7
BuildRequires: rapidjson-devel
%endif
BuildRequires: snappy-devel
BuildRequires: zlib-devel
-%if %{use_python}
-BuildRequires: python34-devel
-BuildRequires: python34-numpy
-%endif
+
%if %{use_parquet}
BuildRequires: bison
BuildRequires: flex
BuildRequires: thrift-devel
%endif
+
%if %{use_glib}
BuildRequires: autoconf-archive
BuildRequires: gtk-doc
@@ -77,20 +82,16 @@ build_type=release
mkdir cpp/build
cd cpp/build
%cmake3 .. \
- -DCMAKE_BUILD_TYPE=$build_type \
-%if %{use_python}
- -DARROW_PYTHON=ON \
- -DPythonInterp_FIND_VERSION=ON \
- -DPythonInterp_FIND_VERSION_MAJOR=3 \
-%endif
-%if %{_centos_ver} == 6
- -DARROW_BOOST_VENDORED=ON \
-%endif
-DARROW_ORC=ON \
%if %{use_parquet}
-DARROW_PARQUET=ON \
%endif
- -DARROW_PLASMA=ON
+ -DARROW_PLASMA=ON \
+ -DARROW_PYTHON=ON \
+ -DARROW_VERBOSE_THIRDPARTY_BUILD=ON \
+ -DCMAKE_BUILD_TYPE=$build_type \
+ -DPythonInterp_FIND_VERSION=ON \
+ -DPythonInterp_FIND_VERSION_MAJOR=3
make %{?_smp_mflags}
cd -
@@ -126,10 +127,11 @@ cd -
%package libs
Summary: Runtime libraries for Apache Arrow C++
License: Apache-2.0
-Requires: boost-system
-Requires: boost-filesystem
-Requires: boost-regex
%if %{_centos_ver} >= 7
+Requires: boost%{boost_version}-system
+Requires: boost%{boost_version}-filesystem
+Requires: boost%{boost_version}-regex
+Requires: brotli
Requires: double-conversion
%endif
Requires: gflags
@@ -168,12 +170,39 @@ Libraries and header files for Apache Arrow
