[
https://issues.apache.org/jira/browse/ARROW-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309243#comment-17309243
]
Joris Van den Bossche commented on ARROW-12066:
-----------------------------------------------
Ah, yes, {{is_null}} was only added recently. The {{is_valid}} method already
existed before, so you can probably achieve the same with
{{~ds.field("A").is_valid()}}
> [Python] Dataset API seg fault when filtering string column for None
> --------------------------------------------------------------------
>
> Key: ARROW-12066
> URL: https://issues.apache.org/jira/browse/ARROW-12066
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 3.0.0
> Environment: macOS 10.15.7
> Reporter: Thomas Blauth
> Priority: Major
>
> Trying to load a parquet file using the dataset api leads to a segmentation
> fault when filtering string columns for None values.
> Minimal reproducing example:
> {code:python}
> import pyarrow as pa
> import pyarrow.dataset
> import pyarrow.parquet
> import pandas as pd
> path = "./test.parquet"
> df = pd.DataFrame({"A": ("a", "b", None)})
> pa.parquet.write_table(pa.table(df), path)
> ds = pa.dataset.dataset(path, format="parquet")
> filter = pa.dataset.field("A") == pa.dataset.scalar(None)
> table = ds.to_table(filter=filter)
> {code}
> Backtrace:
> {code:bash}
> (lldb) target create "/usr/local/mambaforge/envs/xxx/bin/python"
> Current executable set to '/usr/local/mambaforge/envs/xxx/bin/python'
> (x86_64).
> (lldb) settings set -- target.run-args "./tmp.py"
> (lldb) r
> Process 35235 launched: '/usr/local/mambaforge/envs/xxx/bin/python' (x86_64)
> Process 35235 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
> (code=1, address=0x9)
> frame #0: 0x000000010314be48 libarrow.300.0.0.dylib`arrow::Status
> arrow::VisitScalarInline<arrow::ScalarHashImpl>(arrow::Scalar const&,
> arrow::ScalarHashImpl*) + 104
> libarrow.300.0.0.dylib`arrow::VisitScalarInline<arrow::ScalarHashImpl>:
> -> 0x10314be48 <+104>: cmpb $0x0, 0x9(%rax)
> 0x10314be4c <+108>: je 0x10314c0bc ; <+732>
> 0x10314be52 <+114>: movq 0x10(%rax), %rdi
> 0x10314be56 <+118>: movq 0x20(%rax), %rsi
> Target 0: (python) stopped.
> (lldb) bt
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS
> (code=1, address=0x9)
> * frame #0: 0x000000010314be48 libarrow.300.0.0.dylib`arrow::Status
> arrow::VisitScalarInline<arrow::ScalarHashImpl>(arrow::Scalar const&,
> arrow::ScalarHashImpl*) + 104
> frame #1: 0x000000010314bd4f
> libarrow.300.0.0.dylib`arrow::ScalarHashImpl::AccumulateHashFrom(arrow::Scalar
> const&) + 111
> frame #2: 0x0000000103134bca
> libarrow.300.0.0.dylib`arrow::Scalar::Hash::hash(arrow::Scalar const&) + 42
> frame #3: 0x0000000132fa0ea8
> libarrow_dataset.300.0.0.dylib`arrow::dataset::Expression::hash() const + 264
> frame #4: 0x0000000132fc913c
> libarrow_dataset.300.0.0.dylib`std::__1::__hash_const_iterator<std::__1::__hash_node<arrow::dataset::Expression,
> void*>*> std::__1::__hash_table<arrow::dataset::Expression,
> arrow::dataset::Expression::Hash,
> std::__1::equal_to<arrow::dataset::Expression>,
> std::__1::allocator<arrow::dataset::Expression>
> >::find<arrow::dataset::Expression>(arrow::dataset::Expression const&) const
> + 28
> frame #5: 0x0000000132faca9b
> libarrow_dataset.300.0.0.dylib`arrow::Result<arrow::dataset::Expression>
> arrow::dataset::Modify<arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*)::$_1,
> arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*)::$_9>(arrow::dataset::Expression,
> arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*)::$_1 const&,
> arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*)::$_9 const&) + 123
> frame #6: 0x0000000132fac623
> libarrow_dataset.300.0.0.dylib`arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*) + 131
> frame #7: 0x0000000132fac76d
> libarrow_dataset.300.0.0.dylib`arrow::dataset::Canonicalize(arrow::dataset::Expression,
> arrow::compute::ExecContext*) + 461
> frame #8: 0x0000000132fb00cb
> libarrow_dataset.300.0.0.dylib`arrow::dataset::SimplifyWithGuarantee(arrow::dataset::Expression,
> arrow::dataset::Expression const&)::$_10::operator()() const + 75
> frame #9: 0x0000000132faf6b5
> libarrow_dataset.300.0.0.dylib`arrow::dataset::SimplifyWithGuarantee(arrow::dataset::Expression,
> arrow::dataset::Expression const&) + 517
> frame #10: 0x0000000132f893f8
> libarrow_dataset.300.0.0.dylib`arrow::dataset::Dataset::GetFragments(arrow::dataset::Expression)
> + 88
> frame #11: 0x0000000132f8d25c
> libarrow_dataset.300.0.0.dylib`arrow::dataset::GetFragmentsFromDatasets(std::__1::vector<std::__1::shared_ptr<arrow::dataset::Dataset>,
> std::__1::allocator<std::__1::shared_ptr<arrow::dataset::Dataset> > >
> const&,
> arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr<arrow::dataset::Dataset>)::operator()(std::__1::shared_ptr<arrow::dataset::Dataset>)
> const + 76
> frame #12: 0x0000000132f8cd6c
> libarrow_dataset.300.0.0.dylib`arrow::MapIterator<arrow::dataset::GetFragmentsFromDatasets(std::__1::vector<std::__1::shared_ptr<arrow::dataset::Dataset>,
> std::__1::allocator<std::__1::shared_ptr<arrow::dataset::Dataset> > >
> const&,
> arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr<arrow::dataset::Dataset>),
> std::__1::shared_ptr<arrow::dataset::Dataset>,
> arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment> > >::Next() +
> 316
> frame #13: 0x0000000132f8cb27
> libarrow_dataset.300.0.0.dylib`arrow::Result<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> > >
> arrow::Iterator<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >
> >::Next<arrow::MapIterator<arrow::dataset::GetFragmentsFromDatasets(std::__1::vector<std::__1::shared_ptr<arrow::dataset::Dataset>,
> std::__1::allocator<std::__1::shared_ptr<arrow::dataset::Dataset> > >
> const&,
> arrow::dataset::Expression)::'lambda'(std::__1::shared_ptr<arrow::dataset::Dataset>),
> std::__1::shared_ptr<arrow::dataset::Dataset>,
> arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment> > > >(void*) +
> 39
> frame #14: 0x0000000132f8dcdb
> libarrow_dataset.300.0.0.dylib`arrow::Iterator<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> > >::Next() + 43
> frame #15: 0x0000000132f8d692
> libarrow_dataset.300.0.0.dylib`arrow::FlattenIterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >::Next() + 258
> frame #16: 0x0000000132f8d477
> libarrow_dataset.300.0.0.dylib`arrow::Result<std::__1::shared_ptr<arrow::dataset::Fragment>
> > arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >::Next<arrow::FlattenIterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> > >(void*) + 39
> frame #17: 0x0000000132f8de0b
> libarrow_dataset.300.0.0.dylib`arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >::Next() + 43
> frame #18: 0x0000000132fffe80
> libarrow_dataset.300.0.0.dylib`arrow::MapIterator<arrow::dataset::GetScanTaskIterator(arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >, std::__1::shared_ptr<arrow::dataset::ScanOptions>,
> std::__1::shared_ptr<arrow::dataset::ScanContext>)::'lambda'(std::__1::shared_ptr<arrow::dataset::Fragment>),
> std::__1::shared_ptr<arrow::dataset::Fragment>,
> arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask> > >::Next() +
> 48
> frame #19: 0x0000000132fffd47
> libarrow_dataset.300.0.0.dylib`arrow::Result<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> > >
> arrow::Iterator<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> >
> >::Next<arrow::MapIterator<arrow::dataset::GetScanTaskIterator(arrow::Iterator<std::__1::shared_ptr<arrow::dataset::Fragment>
> >, std::__1::shared_ptr<arrow::dataset::ScanOptions>,
> std::__1::shared_ptr<arrow::dataset::ScanContext>)::'lambda'(std::__1::shared_ptr<arrow::dataset::Fragment>),
> std::__1::shared_ptr<arrow::dataset::Fragment>,
> arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask> > > >(void*) +
> 39
> frame #20: 0x0000000133003dcb
> libarrow_dataset.300.0.0.dylib`arrow::Iterator<arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> > >::Next() + 43
> frame #21: 0x0000000133003782
> libarrow_dataset.300.0.0.dylib`arrow::FlattenIterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> >::Next() + 258
> frame #22: 0x0000000133003567
> libarrow_dataset.300.0.0.dylib`arrow::Result<std::__1::shared_ptr<arrow::dataset::ScanTask>
> > arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> >::Next<arrow::FlattenIterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> > >(void*) + 39
> frame #23: 0x0000000132fd479b
> libarrow_dataset.300.0.0.dylib`arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> >::Next() + 43
> frame #24: 0x0000000132fd44e8
> libarrow_dataset.300.0.0.dylib`arrow::Iterator<std::__1::shared_ptr<arrow::dataset::ScanTask>
> >::RangeIterator::Next() + 88
> frame #25: 0x0000000132ffe43d
> libarrow_dataset.300.0.0.dylib`arrow::dataset::Scanner::ToTable() + 589
> frame #26: 0x0000000132f2963a
> _dataset.cpython-39-darwin.so`__pyx_pw_7pyarrow_8_dataset_7Scanner_13to_table(_object*,
> _object*) + 74
> frame #27: 0x0000000132ef47d4
> _dataset.cpython-39-darwin.so`__Pyx_PyObject_CallNoArg(_object*) + 132
> frame #28: 0x0000000132ef0cc9
> _dataset.cpython-39-darwin.so`__pyx_pw_7pyarrow_8_dataset_7Dataset_14to_table(_object*,
> _object*, _object*) + 569
> frame #29: 0x00000001000d5a04 python`cfunction_call + 52
> frame #30: 0x0000000100074998 python`_PyObject_MakeTpCall + 136
> frame #31: 0x00000001001aa8f3 python`call_function + 323
> frame #32: 0x00000001001a843f python`_PyEval_EvalFrameDefault + 45039
> frame #33: 0x000000010019bc04 python`_PyEval_EvalCode + 548
> frame #34: 0x000000010020ec51 python`pyrun_file + 321
> frame #35: 0x000000010020e49c python`pyrun_simple_file + 412
> frame #36: 0x000000010020e2ad python`PyRun_SimpleFileExFlags + 109
> frame #37: 0x0000000100239ed9 python`pymain_run_file + 329
> frame #38: 0x00000001002395c0 python`pymain_run_python + 992
> frame #39: 0x0000000100239185 python`Py_RunMain + 37
> frame #40: 0x000000010023a8f1 python`pymain_main + 49
> frame #41: 0x0000000100001b48 python`main + 56
> frame #42: 0x00007fff73ab2cc9 libdyld.dylib`start + 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
