[jira] [Created] (ARROW-94) Expand list example to clarify null vs empty list
Micah Kornfield created ARROW-94: Summary: Expand list example to clarify null vs empty list Key: ARROW-94 URL: https://issues.apache.org/jira/browse/ARROW-94 Project: Apache Arrow Issue Type: Bug Components: Format Reporter: Micah Kornfield The current spec seems to make the possibility of an empty list impossible (or at least conflates it with null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ARROW-94) Expand list example to clarify null vs empty list
[ https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223286#comment-15223286 ] Wes McKinney commented on ARROW-94: --- The only difference is the null bit -- you must propagate the current offset even with a stretch of nulls so that random access is preserved . Best to clarify in the spec as you aren't the first person to bring this up. > Expand list example to clarify null vs empty list > - > > Key: ARROW-94 > URL: https://issues.apache.org/jira/browse/ARROW-94 > Project: Apache Arrow > Issue Type: Bug > Components: Format >Reporter: Micah Kornfield > > The current spec seems to make the possibility of an empty list impossible > (or at least conflates it with null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ARROW-85) C++: memcmp can be avoided in Equal when comparing with the same Buffer
[ https://issues.apache.org/jira/browse/ARROW-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223572#comment-15223572 ] Kai Zheng commented on ARROW-85: https://github.com/apache/arrow/pull/57 > C++: memcmp can be avoided in Equal when comparing with the same Buffer > --- > > Key: ARROW-85 > URL: https://issues.apache.org/jira/browse/ARROW-85 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Kai Zheng > > It looks too expensive to use memcmp to compare two buffers. Instead, the > starting address and length/capacity would be good enough to use. Higher > level codes relying on memcmp behaviour can be done in higher level. > Update: memcmp should be avoided in Equal when comparing with the same > Buffer. In other cases, it's still needed to know the content are the same or > not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ARROW-86) Python: Implement zero-copy Arrow-to-Pandas conversion
[
https://issues.apache.org/jira/browse/ARROW-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney resolved ARROW-86.
---
Resolution: Fixed
Issue resolved by pull request 52
[https://github.com/apache/arrow/pull/52]
> Python: Implement zero-copy Arrow-to-Pandas conversion
> --
>
> Key: ARROW-86
> URL: https://issues.apache.org/jira/browse/ARROW-86
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Uwe L. Korn
>
> For this to work, we need:
> * a check if the Arrow Array could be represented as Pandas Series using the
> same memory buffer
> * a {{numpy.ndarray}} subclass that keeps a reference to the Arrow Array
> With the NumPy C-API it is already possible to create a {{numpy.ndarray}}
> instance that uses the Arrow Array memory buffer internally but there is no
> way to keep track if the {{numpy.ndarray}} still lives or the underlying
> memory can be free'd.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARROW-94) Expand list example to clarify null vs empty list
[ https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223529#comment-15223529 ] Micah Kornfield commented on ARROW-94: -- I agree, you can piece it together from the spec, but it would be good to be a little bit more explicit. I can try to update spec, the one thing that might be difficult is updating the images, are there source files for them or are they hand-drawn? > Expand list example to clarify null vs empty list > - > > Key: ARROW-94 > URL: https://issues.apache.org/jira/browse/ARROW-94 > Project: Apache Arrow > Issue Type: Bug > Components: Format >Reporter: Micah Kornfield > > The current spec seems to make the possibility of an empty list impossible > (or at least conflates it with null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ARROW-94) Expand list example to clarify null vs empty list
[ https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield reassigned ARROW-94: Assignee: Micah Kornfield > Expand list example to clarify null vs empty list > - > > Key: ARROW-94 > URL: https://issues.apache.org/jira/browse/ARROW-94 > Project: Apache Arrow > Issue Type: Bug > Components: Format >Reporter: Micah Kornfield >Assignee: Micah Kornfield > > The current spec seems to make the possibility of an empty list impossible > (or at least conflates it with null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ARROW-94) Expand list example to clarify null vs empty list
[
https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223544#comment-15223544
]
Wes McKinney commented on ARROW-94:
---
There are OmniGraffle files I can send you. I'm thinking it might be better to
transition to plain text "illustrations", i.e.
{code}
data: ["foo", "", null, null, "bar2"]
List
bitmap 00010011
length: 5
null_count: 2
offsets: 0 3 3 3 3 7
children:
- Int32
-
length: 7
null_count: 0
values: f o o b a r 2
{code}
> Expand list example to clarify null vs empty list
> -
>
> Key: ARROW-94
> URL: https://issues.apache.org/jira/browse/ARROW-94
> Project: Apache Arrow
> Issue Type: Bug
> Components: Format
>Reporter: Micah Kornfield
>Assignee: Micah Kornfield
>
> The current spec seems to make the possibility of an empty list impossible
> (or at least conflates it with null).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (ARROW-94) Expand list example to clarify null vs empty list
[
https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223544#comment-15223544
]
Wes McKinney edited comment on ARROW-94 at 4/3/16 11:05 PM:
There are OmniGraffle files I can send you. I'm thinking it might be better to
transition to plain text "illustrations", i.e.
{code}
data ["foo", "", null, null, "bar2"]
List
bitmap 00010011
length: 5
null_count: 2
offsets: 0 3 3 3 3 7
children:
- uint8 (char)
-
length: 7
null_count: 0
values: f o o b a r 2
{code}
was (Author: wesmckinn):
There are OmniGraffle files I can send you. I'm thinking it might be better to
transition to plain text "illustrations", i.e.
{code}
data ["foo", "", null, null, "bar2"]
List
bitmap 00010011
length: 5
null_count: 2
offsets: 0 3 3 3 3 7
children:
- Int32
-
length: 7
null_count: 0
values: f o o b a r 2
{code}
> Expand list example to clarify null vs empty list
> -
>
> Key: ARROW-94
> URL: https://issues.apache.org/jira/browse/ARROW-94
> Project: Apache Arrow
> Issue Type: Bug
> Components: Format
>Reporter: Micah Kornfield
>Assignee: Micah Kornfield
>
> The current spec seems to make the possibility of an empty list impossible
> (or at least conflates it with null).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARROW-94) Expand list example to clarify null vs empty list
[ https://issues.apache.org/jira/browse/ARROW-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223648#comment-15223648 ] Micah Kornfield commented on ARROW-94: -- Sounds good, I will take a pass through and do the conversion as part of the PR for this item. I might also, try to add some hyperlinks back to definitions/salient points so new readers won't have to cache as much in there head. > Expand list example to clarify null vs empty list > - > > Key: ARROW-94 > URL: https://issues.apache.org/jira/browse/ARROW-94 > Project: Apache Arrow > Issue Type: Bug > Components: Format >Reporter: Micah Kornfield >Assignee: Micah Kornfield > > The current spec seems to make the possibility of an empty list impossible > (or at least conflates it with null). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
