[arrow] branch master updated: ARROW-5456: [GLib][Plasma] Fix dependency order on building document
This is an automated email from the ASF dual-hosted git repository. shiro pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new 8e0e2c0 ARROW-5456: [GLib][Plasma] Fix dependency order on building document 8e0e2c0 is described below commit 8e0e2c06cbb797966f66fd2bcf9043c01006be0d Author: Sutou Kouhei AuthorDate: Sat Jun 1 11:38:00 2019 +0900 ARROW-5456: [GLib][Plasma] Fix dependency order on building document We use installed plasma-glib with allow_glib -> plasma_glib order. We must use building plasma-glib instead of installed plasma-glib. Author: Sutou Kouhei Closes #4425 from kou/glib-plasma-fix-dependency-order and squashes the following commits: b8053c19 Fix dependency order on building document --- c_glib/doc/plasma-glib/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/c_glib/doc/plasma-glib/meson.build b/c_glib/doc/plasma-glib/meson.build index 9efc53b..490b650 100644 --- a/c_glib/doc/plasma-glib/meson.build +++ b/c_glib/doc/plasma-glib/meson.build @@ -53,8 +53,8 @@ source_directories = [ join_paths(meson.build_root(), 'plasma-glib'), ] dependencies = [ - arrow_glib, plasma_glib, + arrow_glib, ] if arrow_cuda.found() dependencies += [arrow_cuda_glib]
[arrow] branch master updated: ARROW-4159: [C++] Build with -Wdocumentation when using clang and BUILD_WARNING_LEVEL=CHECKIN
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new a6da5e9 ARROW-4159: [C++] Build with -Wdocumentation when using clang
and BUILD_WARNING_LEVEL=CHECKIN
a6da5e9 is described below
commit a6da5e98196a3bda68613d0cf9e2b8bcda9fd6c8
Author: Wes McKinney
AuthorDate: Fri May 31 16:30:07 2019 -0500
ARROW-4159: [C++] Build with -Wdocumentation when using clang and
BUILD_WARNING_LEVEL=CHECKIN
This will help prevent Doxygen-related build failures
Author: Wes McKinney
Closes #4441 from wesm/ARROW-4159 and squashes the following commits:
a019c1484 Build with -Wdocumentation on clang
---
cpp/cmake_modules/SetupCxxFlags.cmake | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/cpp/cmake_modules/SetupCxxFlags.cmake
b/cpp/cmake_modules/SetupCxxFlags.cmake
index 7c7e7b5..0c07a55 100644
--- a/cpp/cmake_modules/SetupCxxFlags.cmake
+++ b/cpp/cmake_modules/SetupCxxFlags.cmake
@@ -137,14 +137,14 @@ if("${BUILD_WARNING_LEVEL}" STREQUAL "CHECKIN")
#
https://docs.microsoft.com/en-us/cpp/error-messages/compiler-warnings/compiler-warnings-by-compiler-version
set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} /W3 /wd4365 /wd4267 /wd4838")
elseif("${COMPILER_FAMILY}" STREQUAL "clang")
-set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Weverything -Wno-c++98-compat \
+set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -Weverything -Wdocumentation \
+-Wno-c++98-compat \
-Wno-c++98-compat-pedantic -Wno-deprecated -Wno-weak-vtables -Wno-padded \
-Wno-comma -Wno-unused-macros -Wno-unused-parameter -Wno-unused-template
-Wno-undef \
-Wno-shadow -Wno-switch-enum -Wno-exit-time-destructors \
-Wno-global-constructors -Wno-weak-template-vtables
-Wno-undefined-reinterpret-cast \
-Wno-implicit-fallthrough -Wno-unreachable-code-return \
-Wno-float-equal -Wno-missing-prototypes -Wno-documentation-unknown-command \
--Wno-documentation \
-Wno-old-style-cast -Wno-covered-switch-default \
-Wno-cast-align -Wno-vla-extension -Wno-shift-sign-overflow \
-Wno-used-but-marked-unused -Wno-missing-variable-declarations \
[arrow] branch master updated: ARROW-5396: [JS] Support files and streams with no record batches
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 17e0198 ARROW-5396: [JS] Support files and streams with no record
batches
17e0198 is described below
commit 17e0198253f1b528fe7da494100f2861648688c2
Author: ptaylor
AuthorDate: Fri May 31 16:02:20 2019 -0500
ARROW-5396: [JS] Support files and streams with no record batches
Re: #3871, [ARROW-2119](https://issues.apache.org/jira/browse/ARROW-2119),
and closes [ARROW-5396](https://issues.apache.org/jira/browse/ARROW-5396).
This PR updates the JS Readers and Writers to support files and streams
with no RecordBatches. The approach here is two-fold:
1. If the Readers' source message stream terminates after reading the
Schema message, the Reader will yield a dummy zero-length RecordBatch with the
schema.
2. The Writer always writes the schema for any RecordBatch, but skips
writing the RecordBatch field metadata if it's empty.
This is necessary because the reader and writer don't know about each other
when they're communicating via the Node and DOM stream i/o primitives; they
only know about the values pushed through the streams. Since the
RecordBatchReader and Writer don't yield the Schema message as a standalone
value, we pump the stream with a zero-length RecordBatch that contains the
schema instead.
Author: ptaylor
Author: Wes McKinney
Closes #4373 from trxcllnt/js/fix-no-record-batches and squashes the
following commits:
c86069634 Run no-batches integration test for JS also
86d192d5b define an _InternalEmptyRecordBatch class to signal
that the reader source stream has no RecordBatches
193b08de5 ensure reader and writer support the case where a
stream or file has a schema but no recordbatches
---
integration/integration_test.py | 1 -
js/src/ipc/message.ts | 10 --
js/src/ipc/reader.ts| 10 +-
js/src/ipc/writer.ts| 6 --
js/src/recordbatch.ts | 11 +++
5 files changed, 28 insertions(+), 10 deletions(-)
diff --git a/integration/integration_test.py b/integration/integration_test.py
index 0a508e8..0c5aee4 100644
--- a/integration/integration_test.py
+++ b/integration/integration_test.py
@@ -1031,7 +1031,6 @@ def get_generated_json_files(tempdir=None, flight=False):
file_objs = [
(generate_primitive_case([], name='primitive_no_batches')
- .skip_category('JS')
.skip_category('Java')),
generate_primitive_case([17, 20], name='primitive'),
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
diff --git a/js/src/ipc/message.ts b/js/src/ipc/message.ts
index 194e4ac..c8d3b76 100644
--- a/js/src/ipc/message.ts
+++ b/js/src/ipc/message.ts
@@ -165,21 +165,19 @@ export class JSONMessageReader extends MessageReader {
this._json = source instanceof ArrowJSON ? source : new
ArrowJSON(source);
}
public next() {
-const { _json, _batchIndex, _dictionaryIndex } = this;
-const numBatches = _json.batches.length;
-const numDictionaries = _json.dictionaries.length;
+const { _json } = this;
if (!this._schema) {
this._schema = true;
const message = Message.fromJSON(_json.schema,
MessageHeader.Schema);
-return { value: message, done: _batchIndex >= numBatches &&
_dictionaryIndex >= numDictionaries };
+return { done: false, value: message };
}
-if (_dictionaryIndex < numDictionaries) {
+if (this._dictionaryIndex < _json.dictionaries.length) {
const batch = _json.dictionaries[this._dictionaryIndex++];
this._body = batch['data']['columns'];
const message = Message.fromJSON(batch,
MessageHeader.DictionaryBatch);
return { done: false, value: message };
}
-if (_batchIndex < numBatches) {
+if (this._batchIndex < _json.batches.length) {
const batch = _json.batches[this._batchIndex++];
this._body = batch['columns'];
const message = Message.fromJSON(batch, MessageHeader.RecordBatch);
diff --git a/js/src/ipc/reader.ts b/js/src/ipc/reader.ts
index 78c4036..6778c50 100644
--- a/js/src/ipc/reader.ts
+++ b/js/src/ipc/reader.ts
@@ -22,12 +22,12 @@ import { Footer } from './metadata/file';
import { Schema, Field } from '../schema';
import streamAdapters from '../io/adapters';
import { Message } from './metadata/message';
-import { RecordBatch } from '../recordbatch';
import * as metadata from './metadata/message';
import { ArrayBufferViewInput } from '../util/buffer';
import { ByteStream, AsyncByteStream } from '../io/stream';
import { RandomAccessFile, AsyncRandomAccessFile } from '../io/file';
import {
[arrow] branch master updated: ARROW-5433: [C++][Parquet] Improve parquet-reader columns information, strip trailing whitespace from test case
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 0054ef3 ARROW-5433: [C++][Parquet] Improve parquet-reader columns
information, strip trailing whitespace from test case
0054ef3 is described below
commit 0054ef3bbd424f0a87527090d15e2ff953ee7d83
Author: Renat Valiullin
AuthorDate: Fri May 31 13:52:15 2019 -0500
ARROW-5433: [C++][Parquet] Improve parquet-reader columns information,
strip trailing whitespace from test case
Author: Renat Valiullin
Closes #4403 from rip-nsk/ARROW-5433 and squashes the following commits:
2daadd94c replace column name by column path and better
type information
---
cpp/src/parquet/printer.cc | 18 +++--
cpp/src/parquet/reader-test.cc | 161 -
2 files changed, 92 insertions(+), 87 deletions(-)
diff --git a/cpp/src/parquet/printer.cc b/cpp/src/parquet/printer.cc
index 2e44d90..6e49753 100644
--- a/cpp/src/parquet/printer.cc
+++ b/cpp/src/parquet/printer.cc
@@ -84,17 +84,24 @@ void ParquetFilePrinter::DebugPrint(std::ostream& stream,
std::list selecte
stream << "Number of Selected Columns: " << selected_columns.size() << "\n";
for (auto i : selected_columns) {
const ColumnDescriptor* descr = file_metadata->schema()->Column(i);
-stream << "Column " << i << ": " << descr->name() << " ("
- << TypeToString(descr->physical_type()) << ")" << std::endl;
+stream << "Column " << i << ": " << descr->path()->ToDotString() << " ("
+ << TypeToString(descr->physical_type());
+if (descr->logical_type() != LogicalType::NONE) {
+ stream << "/" << LogicalTypeToString(descr->logical_type());
+}
+if (descr->logical_type() == LogicalType::DECIMAL) {
+ stream << "(" << descr->type_precision() << "," << descr->type_scale()
<< ")";
+}
+stream << ")" << std::endl;
}
for (int r = 0; r < file_metadata->num_row_groups(); ++r) {
-stream << "--- Row Group " << r << " ---\n";
+stream << "--- Row Group: " << r << " ---\n";
auto group_reader = fileReader->RowGroup(r);
std::unique_ptr group_metadata =
file_metadata->RowGroup(r);
-stream << "--- Total Bytes " << group_metadata->total_byte_size() << "
---\n";
+stream << "--- Total Bytes: " << group_metadata->total_byte_size() << "
---\n";
stream << "--- Rows: " << group_metadata->num_rows() << " ---\n";
// Print column metadata
@@ -153,7 +160,7 @@ void ParquetFilePrinter::DebugPrint(std::ostream& stream,
std::list selecte
snprintf(buffer, bufsize, "%-*s", COL_WIDTH,
file_metadata->schema()->Column(i)->name().c_str());
- stream << buffer;
+ stream << buffer << '|';
}
if (format_dump) {
continue;
@@ -167,6 +174,7 @@ void ParquetFilePrinter::DebugPrint(std::ostream& stream,
std::list selecte
if (scanner->HasNext()) {
hasRow = true;
scanner->PrintNext(stream, COL_WIDTH);
+ stream << '|';
}
}
stream << "\n";
diff --git a/cpp/src/parquet/reader-test.cc b/cpp/src/parquet/reader-test.cc
index 80316f0..e68052e 100644
--- a/cpp/src/parquet/reader-test.cc
+++ b/cpp/src/parquet/reader-test.cc
@@ -230,98 +230,95 @@ TEST(TestFileReaderAdHoc, NationDictTruncatedDataPage) {
}
TEST(TestDumpWithLocalFile, DumpOutput) {
- std::stringstream ssValues, ssDump;
+ std::string header_output = R"###(File Name: nested_lists.snappy.parquet
+Version: 1.0
+Created By: parquet-mr version 1.8.2 (build
c6522788629e590a53eb79874b95f6c3ff11f16c)
+Total rows: 3
+Number of RowGroups: 1
+Number of Real Columns: 2
+Number of Columns: 2
+Number of Selected Columns: 2
+Column 0: a.list.element.list.element.list.element (BYTE_ARRAY/UTF8)
+Column 1: b (INT32)
+--- Row Group: 0 ---
+--- Total Bytes: 155 ---
+--- Rows: 3 ---
+Column 0
+ Values: 18 Statistics Not Set
+ Compression: SNAPPY, Encodings: RLE PLAIN_DICTIONARY
+ Uncompressed Size: 103, Compressed Size: 104
+Column 1
+ Values: 3, Null Values: 0, Distinct Values: 0
+ Max: 1, Min: 1
+ Compression: SNAPPY, Encodings: BIT_PACKED PLAIN_DICTIONARY
+ Uncompressed Size: 52, Compressed Size: 56
+)###";
+ std::string values_output = R"###(--- Values ---
+element |b |
+a |1 |
+b |1 |
+c |1 |
+NULL |
+d |
+a |
+b |
+c |
+d |
+NULL |
+e |
+a |
+b |
+c
[arrow] branch master updated: ARROW-3294: [C++][Flight] Support Flight on Windows
This is an automated email from the ASF dual-hosted git repository. wesm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new dbeab70 ARROW-3294: [C++][Flight] Support Flight on Windows dbeab70 is described below commit dbeab70863c4d0cd3da800f18f7e474da624bb5f Author: Antoine Pitrou AuthorDate: Fri May 31 10:29:18 2019 -0500 ARROW-3294: [C++][Flight] Support Flight on Windows Author: Antoine Pitrou Closes #4410 from pitrou/ARROW-3294-flight-windows and squashes the following commits: bd4979b33 ARROW-3294: Support Flight on Windows --- appveyor.yml | 1 + ci/appveyor-cpp-setup.bat | 9 +++ ci/cpp-msvc-build-main.bat | 9 ++- cpp/cmake_modules/ThirdpartyToolchain.cmake| 9 +++ cpp/src/arrow/flight/CMakeLists.txt| 20 +- cpp/src/arrow/flight/client.cc | 4 +- cpp/src/arrow/flight/client.h | 8 +-- cpp/src/arrow/flight/client_auth.h | 8 +-- cpp/src/arrow/flight/customize_protobuf.h | 12 cpp/src/arrow/flight/flight-test.cc| 32 ++--- cpp/src/arrow/flight/internal.cc | 2 +- cpp/src/arrow/flight/internal.h| 13 ++-- .../windows_compatibility.h => flight/platform.h} | 23 +++ cpp/src/arrow/flight/protocol-internal.h | 3 + cpp/src/arrow/flight/serialization-internal.cc | 3 +- cpp/src/arrow/flight/server.cc | 68 +++ cpp/src/arrow/flight/server.h | 30 ++--- cpp/src/arrow/flight/server_auth.h | 10 +-- cpp/src/arrow/flight/test-util.cc | 17 + cpp/src/arrow/flight/test-util.h | 49 ++ cpp/src/arrow/flight/types.h | 32 - .../visibility.h} | 37 +++ cpp/src/arrow/io/mman.h| 7 -- cpp/src/arrow/python/flight.cc | 11 ++-- cpp/src/arrow/util/io-util.cc | 76 ++ cpp/src/arrow/util/io-util.h | 39 +++ cpp/src/arrow/util/windows_compatibility.h | 11 +++- python/CMakeLists.txt | 2 +- python/pyarrow/tests/test_flight.py| 7 +- 29 files changed, 411 insertions(+), 141 deletions(-) diff --git a/appveyor.yml b/appveyor.yml index 74c68df..3e0e645 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -65,6 +65,7 @@ environment: - JOB: "Toolchain" GENERATOR: Visual Studio 14 2015 Win64 CONFIGURATION: "Release" + ARROW_BUILD_FLIGHT: "ON" ARROW_BUILD_GANDIVA: "ON" # NOTE: Since ARROW-5403 we have disabled the static CRT build # - JOB: "Static_Crt_Build" diff --git a/ci/appveyor-cpp-setup.bat b/ci/appveyor-cpp-setup.bat index 0f5868a..aa29498 100644 --- a/ci/appveyor-cpp-setup.bat +++ b/ci/appveyor-cpp-setup.bat @@ -17,6 +17,15 @@ @echo on +@rem Avoid picking up AppVeyor-installed OpenSSL (linker errors with gRPC) +@rem XXX Perhaps there is a smarter way of solving this issue? +rd /s /q C:\OpenSSL-Win32 +rd /s /q C:\OpenSSL-Win64 +rd /s /q C:\OpenSSL-v11-Win32 +rd /s /q C:\OpenSSL-v11-Win64 +rd /s /q C:\OpenSSL-v111-Win32 +rd /s /q C:\OpenSSL-v111-Win64 + conda update -y -q conda conda config --set auto_update_conda false conda info -a diff --git a/ci/cpp-msvc-build-main.bat b/ci/cpp-msvc-build-main.bat index b3ac9bb..80a4cca 100644 --- a/ci/cpp-msvc-build-main.bat +++ b/ci/cpp-msvc-build-main.bat @@ -61,6 +61,7 @@ cmake -G "%GENERATOR%" %CMAKE_ARGS% ^ -DARROW_VERBOSE_THIRDPARTY_BUILD=ON ^ -DARROW_CXXFLAGS="%ARROW_CXXFLAGS%" ^ -DCMAKE_CXX_FLAGS_RELEASE="/MD %CMAKE_CXX_FLAGS_RELEASE%" ^ + -DARROW_FLIGHT=%ARROW_BUILD_FLIGHT% ^ -DARROW_GANDIVA=%ARROW_BUILD_GANDIVA% ^ -DARROW_PARQUET=ON ^ -DPARQUET_BUILD_EXECUTABLES=ON ^ @@ -87,10 +88,16 @@ pip install -r requirements.txt pickle5 set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS% set PYARROW_CMAKE_GENERATOR=%GENERATOR% -set PYARROW_BUNDLE_ARROW_CPP=ON +if "%ARROW_BUILD_FLIGHT%" == "ON" ( + @rem ARROW-5441: bundling Arrow Flight libraries not implemented + set PYARROW_BUNDLE_ARROW_CPP=OFF +) else ( + set PYARROW_BUNDLE_ARROW_CPP=ON +) set PYARROW_BUNDLE_BOOST=OFF set PYARROW_WITH_STATIC_BOOST=ON set PYARROW_WITH_PARQUET=ON +set PYARROW_WITH_FLIGHT=%ARROW_BUILD_FLIGHT% set PYARROW_WITH_GANDIVA=%ARROW_BUILD_GANDIVA% set PYARROW_PARALLEL=2 diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake index 33405c5..5f249c3 100644 --- a/cpp/cmake_modules/ThirdpartyToolchain.cmake +++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake @@ -1075,6 +1075,10 @@
[arrow] branch master updated: ARROW-5462: [Go] support writing zero-length List arrays
This is an automated email from the ASF dual-hosted git repository.
sbinet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new e61bd90 ARROW-5462: [Go] support writing zero-length List arrays
e61bd90 is described below
commit e61bd90153f61a1b627e693a377f06636d2718bf
Author: Sebastien Binet
AuthorDate: Fri May 31 13:45:16 2019 +0200
ARROW-5462: [Go] support writing zero-length List arrays
Author: Sebastien Binet
Closes #4433 from sbinet/issue-5462 and squashes the following commits:
9637f35f ARROW-5462: support writing zero-length List
arrays
---
go/arrow/array/record.go| 5 ++---
go/arrow/array/table.go | 2 +-
go/arrow/internal/arrdata/arrdata.go| 8
go/arrow/ipc/cmd/arrow-cat/main_test.go | 12 +---
go/arrow/ipc/cmd/arrow-ls/main_test.go | 6 +++---
go/arrow/ipc/file_test.go | 1 -
go/arrow/ipc/writer.go | 15 ---
7 files changed, 35 insertions(+), 14 deletions(-)
diff --git a/go/arrow/array/record.go b/go/arrow/array/record.go
index ac70b9c..7bbdab7 100644
--- a/go/arrow/array/record.go
+++ b/go/arrow/array/record.go
@@ -18,7 +18,6 @@ package array
import (
"fmt"
- "reflect"
"sync/atomic"
"github.com/apache/arrow/go/arrow"
@@ -187,10 +186,10 @@ func (rec *simpleRecord) validate() error {
arr.Len(), rec.rows,
)
}
- if !reflect.DeepEqual(f.Type, arr.DataType()) {
+ if !arrow.TypeEquals(f.Type, arr.DataType()) {
return fmt.Errorf("arrow/array: column %q type
mismatch: got=%v, want=%v",
f.Name,
- arr.DataType().Name(), f.Type.Name(),
+ arr.DataType(), f.Type,
)
}
}
diff --git a/go/arrow/array/table.go b/go/arrow/array/table.go
index ff2c8ba..4d8bb45 100644
--- a/go/arrow/array/table.go
+++ b/go/arrow/array/table.go
@@ -55,7 +55,7 @@ func NewColumn(field arrow.Field, chunks *Chunked) *Column {
}
col.data.Retain()
- if col.data.DataType() != col.field.Type {
+ if !arrow.TypeEquals(col.data.DataType(), col.field.Type) {
col.data.Release()
panic("arrow/array: inconsistent data type")
}
diff --git a/go/arrow/internal/arrdata/arrdata.go
b/go/arrow/internal/arrdata/arrdata.go
index c3ee8f1..0df3348 100644
--- a/go/arrow/internal/arrdata/arrdata.go
+++ b/go/arrow/internal/arrdata/arrdata.go
@@ -203,6 +203,14 @@ func makeListsRecords() []array.Record {
arrayOf(mem, []int32{-21, -22, -23, -24, -25},
mask),
}, []bool{true, false, true}),
},
+ []array.Interface{
+ func() array.Interface {
+ bldr := array.NewListBuilder(mem,
arrow.PrimitiveTypes.Int32)
+ defer bldr.Release()
+
+ return bldr.NewListArray()
+ }(),
+ },
}
defer func() {
diff --git a/go/arrow/ipc/cmd/arrow-cat/main_test.go
b/go/arrow/ipc/cmd/arrow-cat/main_test.go
index 0da306f..1fcd90a 100644
--- a/go/arrow/ipc/cmd/arrow-cat/main_test.go
+++ b/go/arrow/ipc/cmd/arrow-cat/main_test.go
@@ -91,6 +91,8 @@ record 2...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] [-11 (null) (null) -14
-15] [-21 (null) (null) -24 -25]]
record 3...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] (null) [-21 (null) (null)
-24 -25]]
+record 4...
+ col[0] "list_nullable": []
`,
},
{
@@ -276,17 +278,21 @@ record 2...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] [-11 (null) (null) -14
-15] [-21 (null) (null) -24 -25]]
record 3...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] (null) [-21 (null) (null)
-24 -25]]
+record 4...
+ col[0] "list_nullable": []
`,
},
{
name: "lists",
want: `version: V4
-record 1/3...
+record 1/4...
col[0] "list_nullable": [[1 (null) (null) 4 5] [11 (null) (null) 14 15] [21
(null) (null) 24 25]]
-record 2/3...
+record 2/4...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] [-11 (null) (null) -14
-15] [-21 (null) (null) -24 -25]]
-record 3/3...
+record 3/4...
col[0] "list_nullable": [[-1 (null) (null) -4 -5] (null) [-21 (null) (null)
-24 -25]]
+record 4/4...
+ col[0] "list_nullable": []
`,
},
{
diff --git a/go/arrow/ipc/cmd/arrow-ls/main_test.go
b/go/arrow/ipc/cmd/arrow-ls/main_test.go
index 7f77304..7fcccd8 100644
--- a/go/arrow/ipc/cmd/arrow-ls/main_test.go
+++
[arrow] branch master updated: ARROW-5459: [Go] implement Stringer for float16 DataType
This is an automated email from the ASF dual-hosted git repository.
sbinet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 1c50bb4 ARROW-5459: [Go] implement Stringer for float16 DataType
1c50bb4 is described below
commit 1c50bb4f1c67d1ed49954bd171a017b213c2a398
Author: Sebastien Binet
AuthorDate: Fri May 31 10:23:29 2019 +0200
ARROW-5459: [Go] implement Stringer for float16 DataType
Author: Sebastien Binet
Closes #4431 from sbinet/issue-5459 and squashes the following commits:
86ff3f09 ARROW-5459: implement Stringer for float16
DataType
---
go/arrow/datatype_fixedwidth.go | 5 +++--
go/arrow/float16/float16.go | 25 ++---
go/arrow/float16/float16_test.go | 23 ---
go/arrow/type_string.go | 6 +++---
go/arrow/type_traits_float16.go | 2 +-
5 files changed, 37 insertions(+), 24 deletions(-)
diff --git a/go/arrow/datatype_fixedwidth.go b/go/arrow/datatype_fixedwidth.go
index bfd6382..2ca 100644
--- a/go/arrow/datatype_fixedwidth.go
+++ b/go/arrow/datatype_fixedwidth.go
@@ -114,8 +114,9 @@ var (
type Float16Type struct{}
-func (t *Float16Type) ID() Type { return FLOAT16 }
-func (t *Float16Type) Name() string { return "float16" }
+func (t *Float16Type) ID() Type { return FLOAT16 }
+func (t *Float16Type) Name() string { return "float16" }
+func (t *Float16Type) String() string { return "float16" }
// BitWidth returns the number of bits required to store a single element of
this data type in memory.
func (t *Float16Type) BitWidth() int { return 16 }
diff --git a/go/arrow/float16/float16.go b/go/arrow/float16/float16.go
index 0ae2e69..e15ee2c 100644
--- a/go/arrow/float16/float16.go
+++ b/go/arrow/float16/float16.go
@@ -14,15 +14,23 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-package float16
+package float16 // import "github.com/apache/arrow/go/arrow/float16"
import (
"math"
+ "strconv"
)
-type Num struct{ Val uint16 }
+// Num represents a half-precision floating point value (float16)
+// stored on 16 bits.
+//
+// See https://en.wikipedia.org/wiki/Half-precision_floating-point_format for
more informations.
+type Num struct {
+ bits uint16
+}
-// https://en.wikipedia.org/wiki/Half-precision_floating-point_format
+// New creates a new half-precision floating point value from the provided
+// float32 value.
func New(f float32) Num {
b := math.Float32bits(f)
sn := uint16((b >> 31) & 0x1)
@@ -41,14 +49,14 @@ func New(f float32) Num {
res = 0
fc = 0
}
- return Num{Val: (sn << 15) | uint16(res<<10) | fc}
+ return Num{bits: (sn << 15) | uint16(res<<10) | fc}
}
func (f Num) Float32() float32 {
- sn := uint32((f.Val >> 15) & 0x1)
- exp := (f.Val >> 10) & 0x1f
+ sn := uint32((f.bits >> 15) & 0x1)
+ exp := (f.bits >> 10) & 0x1f
res := uint32(exp) + 127 - 15
- fc := uint32(f.Val & 0x3ff)
+ fc := uint32(f.bits & 0x3ff)
switch {
case exp == 0:
res = 0
@@ -57,3 +65,6 @@ func (f Num) Float32() float32 {
}
return math.Float32frombits((sn << 31) | (res << 23) | (fc << 13))
}
+
+func (f Num) Uint16() uint16 { return f.bits }
+func (f Num) String() string { return
strconv.FormatFloat(float64(f.Float32()), 'g', -1, 32) }
diff --git a/go/arrow/float16/float16_test.go b/go/arrow/float16/float16_test.go
index 7552c9f..aa96a2f 100644
--- a/go/arrow/float16/float16_test.go
+++ b/go/arrow/float16/float16_test.go
@@ -1,5 +1,3 @@
-// Code generated by type_test.go.tmpl. DO NOT EDIT.
-
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -19,6 +17,7 @@
package float16
import (
+ "fmt"
"testing"
"github.com/stretchr/testify/assert"
@@ -26,19 +25,21 @@ import (
func TestFloat16(t *testing.T) {
cases := map[Num]float32{
- {Val: 0x3c00}: 1,
- {Val: 0x4000}: 2,
- {Val: 0xc000}: -2,
- {Val: 0x}: 0,
- {Val: 0x5b8f}: 241.875,
- {Val: 0xdb8f}: -241.875,
- {Val: 0x48c8}: 9.5625,
- {Val: 0xc8c8}: -9.5625,
+ {bits: 0x3c00}: 1,
+ {bits: 0x4000}: 2,
+ {bits: 0xc000}: -2,
+ {bits: 0x}: 0,
+ {bits: 0x5b8f}: 241.875,
+ {bits: 0xdb8f}: -241.875,
+ {bits: 0x48c8}: 9.5625,
+ {bits: 0xc8c8}: -9.5625,
}
for k, v := range cases {
f := k.Float32()
assert.Equal(t, v, f, "float32 values should be the same")
[arrow] branch master updated: ARROW-5384: [Go] implement FixedSizeList array
This is an automated email from the ASF dual-hosted git repository.
sbinet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new a7dadb3 ARROW-5384: [Go] implement FixedSizeList array
a7dadb3 is described below
commit a7dadb37afde6d79b309ce583765c11469413916
Author: Sebastien Binet
AuthorDate: Fri May 31 09:22:15 2019 +0200
ARROW-5384: [Go] implement FixedSizeList array
Author: Sebastien Binet
Closes #4357 from sbinet/issue-5384 and squashes the following commits:
68fc185f handle slice-offsets
b7f12be4 ARROW-5384: implement FixedSizeList array
---
go/arrow/array/array.go| 6 +-
go/arrow/array/array_test.go | 14 +-
go/arrow/array/builder.go | 5 +
go/arrow/array/fixed_size_list.go | 244 +
go/arrow/array/fixed_size_list_test.go | 225 ++
go/arrow/datatype.go | 10 ++
go/arrow/datatype_nested.go| 34 +
go/arrow/datatype_nested_test.go | 63 +
go/arrow/example_test.go | 76 ++
go/arrow/internal/arrdata/arrdata.go | 75 ++
go/arrow/ipc/file_reader.go| 16 +++
go/arrow/ipc/metadata.go | 15 ++
go/arrow/ipc/writer.go | 38 +
go/arrow/type_string.go| 41 +-
14 files changed, 856 insertions(+), 6 deletions(-)
diff --git a/go/arrow/array/array.go b/go/arrow/array/array.go
index ef37aef..82a1c01 100644
--- a/go/arrow/array/array.go
+++ b/go/arrow/array/array.go
@@ -192,11 +192,11 @@ func init() {
arrow.UNION: unsupportedArrayType,
arrow.DICTIONARY:unsupportedArrayType,
arrow.MAP: unsupportedArrayType,
+ arrow.EXTENSION: unsupportedArrayType,
+ arrow.FIXED_SIZE_LIST: func(data *Data) Interface { return
NewFixedSizeListData(data) },
+ arrow.DURATION: unsupportedArrayType,
// invalid data types to fill out array size 2⁵-1
- 28: invalidDataType,
- 29: invalidDataType,
- 30: invalidDataType,
31: invalidDataType,
}
}
diff --git a/go/arrow/array/array_test.go b/go/arrow/array/array_test.go
index 014a79a..3a3407f 100644
--- a/go/arrow/array/array_test.go
+++ b/go/arrow/array/array_test.go
@@ -76,9 +76,21 @@ func TestMakeFromData(t *testing.T) {
array.NewData({arrow.INT64}, 0,
make([]*memory.Buffer, 4), nil, 0, 0),
}},
+ {name: "fixed_size_list", d:
{arrow.FIXED_SIZE_LIST}, child: []*array.Data{
+ array.NewData({arrow.INT64}, 0,
make([]*memory.Buffer, 4), nil, 0, 0),
+ array.NewData({arrow.INT64}, 0,
make([]*memory.Buffer, 4), nil, 0, 0),
+ }},
+
+ // unsupported types
+ {name: "union", d: {arrow.UNION}, expPanic: true,
expError: "unsupported data type: UNION"},
+ {name: "dictionary", d: {arrow.DICTIONARY},
expPanic: true, expError: "unsupported data type: DICTIONARY"},
+ {name: "map", d: {arrow.Type(27)}, expPanic: true,
expError: "unsupported data type: MAP"},
+ {name: "extension", d: {arrow.Type(28)}, expPanic:
true, expError: "unsupported data type: EXTENSION"},
+ {name: "duration", d: {arrow.Type(30)}, expPanic:
true, expError: "unsupported data type: DURATION"},
+
// invalid types
{name: "invalid(-1)", d: {arrow.Type(-1)},
expPanic: true, expError: "invalid data type: Type(-1)"},
- {name: "invalid(28)", d: {arrow.Type(28)},
expPanic: true, expError: "invalid data type: Type(28)"},
+ {name: "invalid(31)", d: {arrow.Type(31)},
expPanic: true, expError: "invalid data type: Type(31)"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
diff --git a/go/arrow/array/builder.go b/go/arrow/array/builder.go
index d6cec85..c815cd4 100644
--- a/go/arrow/array/builder.go
+++ b/go/arrow/array/builder.go
@@ -262,6 +262,11 @@ func newBuilder(mem memory.Allocator, dtype
arrow.DataType) Builder {
case arrow.UNION:
case arrow.DICTIONARY:
case arrow.MAP:
+ case arrow.EXTENSION:
+ case arrow.FIXED_SIZE_LIST:
+ typ := dtype.(*arrow.FixedSizeListType)
+ return NewFixedSizeListBuilder(mem, typ.Len(), typ.Elem())
+ case arrow.DURATION:
}
panic(fmt.Errorf("arrow/array: unsupported builder for %T", dtype))
}
diff --git a/go/arrow/array/fixed_size_list.go
b/go/arrow/array/fixed_size_list.go
new file mode 100644
index 000..9816e65
--- /dev/null
+++ b/go/arrow/array/fixed_size_list.go
@@ -0,0
[arrow] branch master updated: ARROW-5387: [Go] properly handle sub-slice of List
This is an automated email from the ASF dual-hosted git repository.
sbinet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 17e85bf ARROW-5387: [Go] properly handle sub-slice of List
17e85bf is described below
commit 17e85bf63a29129fadeb8939cdcd18c8935dd0da
Author: Sebastien Binet
AuthorDate: Fri May 31 09:20:52 2019 +0200
ARROW-5387: [Go] properly handle sub-slice of List
Author: Sebastien Binet
Closes #4360 from sbinet/issue-5387 and squashes the following commits:
bbc8a837 ARROW-5387: properly handle sub-slice of List
---
go/arrow/array/list.go | 5 ++--
go/arrow/array/list_test.go | 62 +
2 files changed, 65 insertions(+), 2 deletions(-)
diff --git a/go/arrow/array/list.go b/go/arrow/array/list.go
index 9b74641..1b91176 100644
--- a/go/arrow/array/list.go
+++ b/go/arrow/array/list.go
@@ -55,8 +55,9 @@ func (a *List) String() string {
o.WriteString("(null)")
continue
}
- beg := int64(a.offsets[i])
- end := int64(a.offsets[i+1])
+ j := i + a.array.data.offset
+ beg := int64(a.offsets[j])
+ end := int64(a.offsets[j+1])
sub := NewSlice(a.values, beg, end)
fmt.Fprintf(o, "%v", sub)
sub.Release()
diff --git a/go/arrow/array/list_test.go b/go/arrow/array/list_test.go
index 0853776..b328cf4 100644
--- a/go/arrow/array/list_test.go
+++ b/go/arrow/array/list_test.go
@@ -149,3 +149,65 @@ func TestListArrayBulkAppend(t *testing.T) {
t.Fatalf("got=%v, want=%v", got, want)
}
}
+
+func TestListArraySlice(t *testing.T) {
+ pool := memory.NewCheckedAllocator(memory.NewGoAllocator())
+ defer pool.AssertSize(t, 0)
+
+ var (
+ vs = []int32{0, 1, 2, 3, 4, 5, 6}
+ lengths = []int{3, 0, 4}
+ isValid = []bool{true, false, true}
+ offsets = []int32{0, 3, 3, 7}
+ )
+
+ lb := array.NewListBuilder(pool, arrow.PrimitiveTypes.Int32)
+ defer lb.Release()
+ vb := lb.ValueBuilder().(*array.Int32Builder)
+ vb.Reserve(len(vs))
+
+ lb.AppendValues(offsets, isValid)
+ for _, v := range vs {
+ vb.Append(v)
+ }
+
+ arr := lb.NewArray().(*array.List)
+ defer arr.Release()
+
+ if got, want := arr.DataType().ID(), arrow.LIST; got != want {
+ t.Fatalf("got=%v, want=%v", got, want)
+ }
+
+ if got, want := arr.Len(), len(isValid); got != want {
+ t.Fatalf("got=%d, want=%d", got, want)
+ }
+
+ for i := range lengths {
+ if got, want := arr.IsValid(i), isValid[i]; got != want {
+ t.Fatalf("got[%d]=%v, want[%d]=%v", i, got, i, want)
+ }
+ if got, want := arr.IsNull(i), lengths[i] == 0; got != want {
+ t.Fatalf("got[%d]=%v, want[%d]=%v", i, got, i, want)
+ }
+ }
+
+ if got, want := arr.Offsets(), offsets; !reflect.DeepEqual(got, want) {
+ t.Fatalf("got=%v, want=%v", got, want)
+ }
+
+ varr := arr.ListValues().(*array.Int32)
+ if got, want := varr.Int32Values(), vs; !reflect.DeepEqual(got, want) {
+ t.Fatalf("got=%v, want=%v", got, want)
+ }
+
+ if got, want := arr.String(), `[[0 1 2] (null) [3 4 5 6]]`; got != want
{
+ t.Fatalf("got=%q, want=%q", got, want)
+ }
+
+ sub := array.NewSlice(arr, 1, 3).(*array.List)
+ defer sub.Release()
+
+ if got, want := sub.String(), `[(null) [3 4 5 6]]`; got != want {
+ t.Fatalf("got=%q, want=%q", got, want)
+ }
+}
[arrow] branch master updated: ARROW-5457: [GLib][Plasma] Fix environment variable name for test
This is an automated email from the ASF dual-hosted git repository.
emkornfield pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 65ae609 ARROW-5457: [GLib][Plasma] Fix environment variable name for
test
65ae609 is described below
commit 65ae6098b6d28771c83c7295086f63749b520831
Author: Sutou Kouhei
AuthorDate: Fri May 31 00:14:32 2019 -0700
ARROW-5457: [GLib][Plasma] Fix environment variable name for test
Author: Sutou Kouhei
Closes #4426 from kou/glib-test-fix-path and squashes the following commits:
884309bf Fix environment variable name for test
---
c_glib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/c_glib/meson.build b/c_glib/meson.build
index 7af2e5d..d3a2441 100644
--- a/c_glib/meson.build
+++ b/c_glib/meson.build
@@ -138,5 +138,5 @@ test('unit test',
'ARROW_CUDA_GLIB_TYPELIB_DIR=@0@/arrow-cuda-glib'.format(meson.build_root()),
'GANDIVA_GLIB_TYPELIB_DIR=@0@/gandiva-glib'.format(meson.build_root()),
'PARQUET_GLIB_TYPELIB_DIR=@0@/parquet-glib'.format(meson.build_root()),
- 'PARQUET_GLIB_TYPELIB_DIR=@0@/plasma-glib'.format(meson.build_root()),
+ 'PLASMA_GLIB_TYPELIB_DIR=@0@/plasma-glib'.format(meson.build_root()),
])
[arrow] branch master updated: ARROW-5455: [Rust] Build broken by 2019-05-30 Rust nightly
This is an automated email from the ASF dual-hosted git repository. kou pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/master by this push: new a40aee6 ARROW-5455: [Rust] Build broken by 2019-05-30 Rust nightly a40aee6 is described below commit a40aee6cad2add60fc7a2360b36d31b2255a5262 Author: Chao Sun AuthorDate: Fri May 31 15:40:27 2019 +0900 ARROW-5455: [Rust] Build broken by 2019-05-30 Rust nightly Temporarily disable the lint on bare trait objects. Will switch to the new "dyn trait" syntax later and re-enable the lint. Author: Chao Sun Closes #4429 from sunchao/ARROW-5455 and squashes the following commits: 05d66771 ARROW-5455: Build broken by 2019-05-30 Rust nightly --- rust/arrow/src/lib.rs | 1 + rust/datafusion/src/bin/repl.rs | 2 ++ rust/datafusion/src/lib.rs | 1 + rust/parquet/src/lib.rs | 1 + 4 files changed, 5 insertions(+) diff --git a/rust/arrow/src/lib.rs b/rust/arrow/src/lib.rs index fb7bec7..a7ef817 100644 --- a/rust/arrow/src/lib.rs +++ b/rust/arrow/src/lib.rs @@ -24,6 +24,7 @@ #![feature(specialization)] #![allow(dead_code)] #![allow(non_camel_case_types)] +#![allow(bare_trait_objects)] pub mod array; pub mod array_data; diff --git a/rust/datafusion/src/bin/repl.rs b/rust/datafusion/src/bin/repl.rs index ac73b26..88a8894 100644 --- a/rust/datafusion/src/bin/repl.rs +++ b/rust/datafusion/src/bin/repl.rs @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +#![allow(bare_trait_objects)] + #[macro_use] extern crate clap; diff --git a/rust/datafusion/src/lib.rs b/rust/datafusion/src/lib.rs index 22805cf..65ff375 100644 --- a/rust/datafusion/src/lib.rs +++ b/rust/datafusion/src/lib.rs @@ -19,6 +19,7 @@ //! Apache Arrow as the memory model #![warn(missing_docs)] +#![allow(bare_trait_objects)] extern crate arrow; #[macro_use] diff --git a/rust/parquet/src/lib.rs b/rust/parquet/src/lib.rs index 1e247d5..8d23e89 100644 --- a/rust/parquet/src/lib.rs +++ b/rust/parquet/src/lib.rs @@ -18,6 +18,7 @@ #![feature(specialization)] #![allow(dead_code)] #![allow(non_camel_case_types)] +#![allow(bare_trait_objects)] #[macro_use] pub mod errors;
