[NIGHTLY] Arrow Build Report for Job nightly-2020-03-28-1
Arrow Build Report for Job nightly-2020-03-28-1 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1 Failed Tasks: - centos-7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-centos-7 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-linux-gcc-py38 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-osx-clang-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-conda-win-vs2015-py38 - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-debian-buster - gandiva-jar-trusty: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-travis-gandiva-jar-trusty - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.6 - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7-spark-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.7-spark-master - test-conda-python-3.7-turbodbc-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.7-turbodbc-master - test-conda-python-3.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.7 - test-conda-python-3.8-jpype: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-conda-python-3.8-jpype - test-r-linux-as-cran: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-test-r-linux-as-cran - test-r-rhub-ubuntu-gcc-release: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rhub-ubuntu-gcc-release - test-r-rocker-r-base-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rocker-r-base-latest - test-r-rstudio-r-base-3.6-bionic: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rstudio-r-base-3.6-bionic - test-r-rstudio-r-base-3.6-centos6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rstudio-r-base-3.6-centos6 - test-r-rstudio-r-base-3.6-opensuse15: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rstudio-r-base-3.6-opensuse15 - test-r-rstudio-r-base-3.6-opensuse42: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-azure-test-r-rstudio-r-base-3.6-opensuse42 - test-ubuntu-16.04-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-ubuntu-16.04-cpp - test-ubuntu-18.04-cpp-cmake32: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-ubuntu-18.04-cpp-cmake32 - test-ubuntu-18.04-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-circle-test-ubuntu-18.04-docs Succeeded Tasks: - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-centos-6 - centos-8: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-centos-8 - debian-stretch: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-github-debian-stretch - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-travis-gandiva-jar-osx - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-travis-homebrew-cpp - macos-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-1-travis-macos-r-autobrew -
[jira] [Created] (ARROW-8252) [CI][Ruby] Add Ubuntu 20.04
Kouhei Sutou created ARROW-8252: --- Summary: [CI][Ruby] Add Ubuntu 20.04 Key: ARROW-8252 URL: https://issues.apache.org/jira/browse/ARROW-8252 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration, Ruby Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[NIGHTLY] Arrow Build Report for Job nightly-2020-03-28-0
Arrow Build Report for Job nightly-2020-03-28-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0 Failed Tasks: - centos-7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-centos-7 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-linux-gcc-py38 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-osx-clang-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-conda-win-vs2015-py38 - gandiva-jar-trusty: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-travis-gandiva-jar-trusty - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-conda-python-3.6 - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-conda-python-3.7 - test-conda-python-3.8-jpype: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-conda-python-3.8-jpype - test-debian-ruby: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-debian-ruby - test-r-linux-as-cran: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-test-r-linux-as-cran - test-r-rhub-ubuntu-gcc-release: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rhub-ubuntu-gcc-release - test-r-rocker-r-base-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rocker-r-base-latest - test-r-rstudio-r-base-3.6-bionic: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rstudio-r-base-3.6-bionic - test-r-rstudio-r-base-3.6-centos6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rstudio-r-base-3.6-centos6 - test-r-rstudio-r-base-3.6-opensuse15: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rstudio-r-base-3.6-opensuse15 - test-r-rstudio-r-base-3.6-opensuse42: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-azure-test-r-rstudio-r-base-3.6-opensuse42 - test-ubuntu-16.04-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-ubuntu-16.04-cpp - test-ubuntu-18.04-cpp-cmake32: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-ubuntu-18.04-cpp-cmake32 - test-ubuntu-18.04-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-ubuntu-18.04-docs Succeeded Tasks: - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-centos-6 - centos-8: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-centos-8 - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-debian-buster - debian-stretch: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-github-debian-stretch - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-travis-gandiva-jar-osx - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-travis-homebrew-cpp - macos-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-travis-macos-r-autobrew - test-conda-cpp-valgrind: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-03-28-0-circle-test-conda-cpp-valgrind - test-conda-cpp: URL: https://github.com/ursa-labs/crossbow/br
Re: [VOTE] Accept "DoExchange" RPC to Arrow Flight protocol
+1 non-binding On Sat, Mar 28, 2020 at 1:44 AM Wes McKinney wrote: > Hello, > > David M Li has proposed adding a "bidirectional" DoExchange RPC [1] to > the Arrow Flight Protocol [2]. In this client call, datasets (possibly > having different schemas) are sent by both the > client and server in a single transaction. This can be used to offload > computational tasks and other workloads not currently well-supported > by the Flight protocol. > > Please vote whether to accept the addition. The vote will be open for > at least 72 hours (since it's Friday, it'll be open for a good deal > longer than 72 hours). > > [ ] +1 Accept this addition to the Flight protocol > [ ] +0 > [ ] -1 Do not accept the changes because... > > Here is my vote: +1 > > Thanks, > Wes > > [1]: https://github.com/apache/arrow/pull/6686 >
Re: [VOTE] Accept "DoExchange" RPC to Arrow Flight protocol
+1 (binding) Le 28/03/2020 à 01:44, Wes McKinney a écrit : > Hello, > > David M Li has proposed adding a "bidirectional" DoExchange RPC [1] to > the Arrow Flight Protocol [2]. In this client call, datasets (possibly > having different schemas) are sent by both the > client and server in a single transaction. This can be used to offload > computational tasks and other workloads not currently well-supported > by the Flight protocol. > > Please vote whether to accept the addition. The vote will be open for > at least 72 hours (since it's Friday, it'll be open for a good deal > longer than 72 hours). > > [ ] +1 Accept this addition to the Flight protocol > [ ] +0 > [ ] -1 Do not accept the changes because... > > Here is my vote: +1 > > Thanks, > Wes > > [1]: https://github.com/apache/arrow/pull/6686 >
[jira] [Created] (ARROW-8251) [Python] pandas.ExtensionDtype does not survive round trip with write_to_dataset
Ged Steponavicius created ARROW-8251:
Summary: [Python] pandas.ExtensionDtype does not survive round
trip with write_to_dataset
Key: ARROW-8251
URL: https://issues.apache.org/jira/browse/ARROW-8251
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.16.0
Environment: pandas 1.0.1
parquet 0.16
Reporter: Ged Steponavicius
write_to_dataset with pandas fields using pandas.ExtensionDtype nullable int or
string produce parquet file which when read back in has different dtypes than
original df
{code:java}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
parquet_dataset = 'partquet_dataset/'
parquet_file = 'test.parquet'
df = pd.DataFrame([{'str_col':'abc','int_col':1,'part':1},
{'str_col':np.nan,'int_col':np.nan,'part':1}])
df['str_col'] = df['str_col'].astype(pd.StringDtype())
df['int_col'] = df['int_col'].astype(pd.Int64Dtype())
table = pa.Table.from_pandas(df)
pq.write_to_dataset(table, root_path=parquet_dataset, partition_cols=['part'] )
pq.write_table(table, where=parquet_file) {code}
write_table handles schema correctly, pandas.ExtensionDtype survive round trip:
{code:java}
pq.read_table(parquet_file).to_pandas().dtypes
str_col string
int_col Int64
part int64 {code}
However, write_to_dataset reverts back to object/float:
{code:java}
pq.read_table(parquet_dataset).to_pandas().dtypes
str_col object
int_col float64
part category {code}
I have also tried writing common metadata at the top-level directory of a
partitioned dataset and then passing metadata to read_table, but results are
the same as without metadata
{code:java}
pq.write_metadata(table.schema, parquet_dataset+'_common_metadata',
version='2.0') meta = pq.read_metadata(parquet_dataset+'_common_metadata')
pq.read_table(parquet_dataset,metadata=meta).to_pandas().dtypes {code}
This also affects pandas to_parquet when partition_cols is specified:
{code:java}
df.to_parquet(path = parquet_dataset, partition_cols=['part'])
pd.read_parquet(parquet_dataset).dtypes
str_col object
int_col float64
part category {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
