[
https://issues.apache.org/jira/browse/ARROW-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206214#comment-16206214
]
ASF GitHub Bot commented on ARROW-1676:
---------------------------------------
Github user wesm commented on the issue:
https://github.com/apache/arrow/pull/1204
@kou what can I do about the GLib build failures?
> [C++] Correctly truncate oversized validity bitmaps when writing Feather
> format
> -------------------------------------------------------------------------------
>
> Key: ARROW-1676
> URL: https://issues.apache.org/jira/browse/ARROW-1676
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 0.7.1
> Environment: libarrow-dev
> Architecture: amd64
> Version: 0.7.1-1
> Python 2.7.13
> >>> pyarrow.__version__
> '0.7.1'
> >>> feather.__version__
> '0.4.0'
> >>> pandas.__version__
> u'0.20.3'
> Reporter: Rares Vernica
> Assignee: Wes McKinney
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> An extra {{0}} appears in the beginning when serializing and deserializing an
> array with more than {{128}} values and at least one {{NULL}} value using
> {{Feather}}. Once the extra {{0}} is inserted a value is trimmed at the end.
> Here is the C++ code to write such an array:
> {code:java}
> #include <iostream>
> #include <arrow/api.h>
> #include <arrow/io/file.h>
> #include <arrow/ipc/feather.h>
> #include <arrow/pretty_print.h>
> int main() {
> // 1. Build Array
> arrow::DoubleBuilder builder;
> for (int i = 0; i < 129; i++)
> if (i == 0)
> builder.AppendNull();
> else
> builder.Append(i);
> std::shared_ptr<arrow::Array> array;
> builder.Finish(&array);
> arrow::PrettyPrint(*array, 0, &std::cout);
> std::cout << std::endl;
> // 2. Write to Feather file
> std::shared_ptr<arrow::io::FileOutputStream> stream;
> arrow::io::FileOutputStream::Open("out.f", false, &stream);
> std::unique_ptr<arrow::ipc::feather::TableWriter> writer;
> arrow::ipc::feather::TableWriter::Open(stream, &writer);
> writer->SetNumRows(129);
> writer->Append("id", *array);
> writer->Finalize();
> stream->Close();
> return 0;
> }
> {code}
> The output of running this code is:
> {code:java}
> # g++-4.9 -std=c++11 example.cpp -larrow && ./a.out
> [null, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
> 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
> 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58,
> 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
> 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96,
> 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
> 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127,
> 128]
> {code}
> The array is deserialized in Python and looks like this:
>
> {code:java}
> >>> pandas.read_feather('out.f')
> id
> 0 NaN
> 1 0.0
> 2 1.0
> 3 2.0
> 4 3.0
> 5 4.0
> 6 5.0
> 7 6.0
> 8 7.0
> 9 8.0
> 10 9.0
> 11 10.0
> 12 11.0
> 13 12.0
> 14 13.0
> 15 14.0
> 16 15.0
> 17 16.0
> 18 17.0
> 19 18.0
> 20 19.0
> 21 20.0
> 22 21.0
> 23 22.0
> 24 23.0
> 25 24.0
> 26 25.0
> 27 26.0
> 28 27.0
> 29 28.0
> .. ...
> 99 98.0
> 100 99.0
> 101 100.0
> 102 101.0
> 103 102.0
> 104 103.0
> 105 104.0
> 106 105.0
> 107 106.0
> 108 107.0
> 109 108.0
> 110 109.0
> 111 110.0
> 112 111.0
> 113 112.0
> 114 113.0
> 115 114.0
> 116 115.0
> 117 116.0
> 118 117.0
> 119 118.0
> 120 119.0
> 121 120.0
> 122 121.0
> 123 122.0
> 124 123.0
> 125 124.0
> 126 125.0
> 127 126.0
> 128 127.0
> [129 rows x 1 columns]
> {code}
> Notice the {{0.0}} value on index {{1}}. The value should have been {{1.0}}.
> Also, the last value is {{127.0}} instead of {{128.0}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
