[jira] [Created] (ARROW-1640) Travis-CI Build Issue
Ankit Aggarwal created ARROW-1640: - Summary: Travis-CI Build Issue Key: ARROW-1640 URL: https://issues.apache.org/jira/browse/ARROW-1640 Project: Apache Arrow Issue Type: Bug Reporter: Ankit Aggarwal Following error occurs: error: downloading 'https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz' failed status_code: 1 status_string: "Unsupported protocol" log: Protocol "https" not supported or disabled in libcurl Closing connection -1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ARROW-1639) [Python] More efficient serialization for RangeIndex in serialize_pandas
Wes McKinney created ARROW-1639: --- Summary: [Python] More efficient serialization for RangeIndex in serialize_pandas Key: ARROW-1639 URL: https://issues.apache.org/jira/browse/ARROW-1639 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Wes McKinney -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Thanks all! It's a pleasure to be working on this project with such a great community. Looking forward to an exciting future with Arrow! On Tue, Oct 3, 2017 at 11:58 AM, Siddharth Teotiawrote: > Congrats Philip and Bryan :) > > On Tue, Oct 3, 2017 at 11:57 AM, Holden Karau > wrote: > > > Congrats to the both of you :) Really excited to see the Areow project > > continue to grow :) > > > > On Tue, Oct 3, 2017 at 10:24 AM Julian Hyde wrote: > > > > > Congratulations and welcome, Philip and Bryan! > > > > > > > On Oct 3, 2017, at 5:27 AM, Wes McKinney > wrote: > > > > > > > > On behalf of the Arrow PMC, I'm pleased to announce that Phillip > Cloud > > > > and Bryan Cutler have been invited to be Arrow committers. > > > > > > > > We are grateful for your contributions to the project and look > forward > > > > to growing the community together. > > > > > > > > Welcome, Phillip and Bryan, and congrats! > > > > > > > > - Wes > > > > > > -- > > Twitter: https://twitter.com/holdenkarau > > >
Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.
Thanks for doing this Siddarth! It sounds like a good idea to post a WIP so we can all take a look and try out some things out. There's a lot going on so it's difficult to try to solve it all on paper, so maybe as we can start with something very rough and take it from there. I think we all agree on the requirements and goals, so that's a good start :) Bryan On Tue, Oct 3, 2017 at 3:58 PM, Siddharth Teotiawrote: > Anyway, I will create a PR in some time for the WIP prototype. I think once > people eyeball the code there, we may have a consensus. > > > On Tue, Oct 3, 2017 at 3:53 PM, Siddharth Teotia > wrote: > > > Li, > > > > This is exactly what I was referring to in my previous email. I think if > > we have the opportunity (which we have now), we should see if complex > > template code like this can be removed unless it is absolutely necessary. > > This is why I was suggesting if we can just not have any code generated > > classes. > > > > Thanks, > > Siddharth > > > > On Tue, Oct 3, 2017 at 3:03 PM, Li Jin wrote: > > > >> Siddharth, > >> > >> Thanks for the update. Without really sit down and do the prototype, my > >> opinions can be wrong. But, > >> > >> I think a lot of the complication are code like this: > >> > >> <#elseif minor.class == "Decimal"> > >> public void get(int index, ${minor.class}Holder holder) { > >> holder.start = index * ${type.width}; > >> holder.buffer = data; > >> holder.scale = scale; > >> holder.precision = precision; > >> } > >> > >> The fact that we have to have a special block for Decimal is eyesore to > >> me. > >> And it's not really shared with any other classes. > >> > >> Things like these are not great but might be ok? > >> > >> <#if type.width == 4> > >> public long getTwoAsLong(int index) { > >> return data.getLong(index * ${type.width}); > >> } > >> > >> > >> > >> Sorry I couldn't provide too much useful feedback without digging into > the > >> template, but this is any general feeling about these templates - too > many > >> "if" to types like "Interval" "Decimal" "Timestamp" > >> > >> > >> > >> On Tue, Oct 3, 2017 at 3:59 PM, Siddharth Teotia > >> wrote: > >> > >> > I am in the middle of a simple prototype that has the basic > >> implementation > >> > of BaseFixedWidthVector, FixedValueVectorsPrototype.java (template) > to > >> > generate a simple IntVector using the proposal mentioned in the > >> document. > >> > > >> > I have realized that even though the LOCs in existing templates are > >> reduced > >> > by 30-40% since bunch of common/basic functionality is moved to super > >> class > >> > BaseFixedWidthVector, the major source of pain (giant and complex if > >> > conditions) associated with code generation is in accessor and mutator > >> > which is still part of templates. > >> > > >> > I am trying to err on the side of not using templates at all since I > >> feel > >> > there is not much of gain from this refactoring project if the code in > >> > templates is still complex and requires regular addition/modification > >> when > >> > adding new types. We are probably better off writing multiple sub > >> classes > >> > (with duplicate code as applicable) > >> > > >> > Thoughts? > >> > > >> > I can create a PR from this prototype code once it in reasonable shape > >> for > >> > review but was wondering if people have any opinion. > >> > > >> > Thanks, > >> > Sidd > >> > > >> > On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotia < > siddha...@dremio.com> > >> > wrote: > >> > > >> > > Hi All, > >> > > > >> > > You should have received an invitation to edit the following > document. > >> > > Please feel free to add comments or additional content. > >> > > > >> > > https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930 > >> > > sBKKaiyggOY7t4/edit > >> > > > >> > > Let me know if the document is not editable. > >> > > > >> > > Thanks, > >> > > Siddharth > >> > > > >> > > >> > > > > >
Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.
Anyway, I will create a PR in some time for the WIP prototype. I think once people eyeball the code there, we may have a consensus. On Tue, Oct 3, 2017 at 3:53 PM, Siddharth Teotiawrote: > Li, > > This is exactly what I was referring to in my previous email. I think if > we have the opportunity (which we have now), we should see if complex > template code like this can be removed unless it is absolutely necessary. > This is why I was suggesting if we can just not have any code generated > classes. > > Thanks, > Siddharth > > On Tue, Oct 3, 2017 at 3:03 PM, Li Jin wrote: > >> Siddharth, >> >> Thanks for the update. Without really sit down and do the prototype, my >> opinions can be wrong. But, >> >> I think a lot of the complication are code like this: >> >> <#elseif minor.class == "Decimal"> >> public void get(int index, ${minor.class}Holder holder) { >> holder.start = index * ${type.width}; >> holder.buffer = data; >> holder.scale = scale; >> holder.precision = precision; >> } >> >> The fact that we have to have a special block for Decimal is eyesore to >> me. >> And it's not really shared with any other classes. >> >> Things like these are not great but might be ok? >> >> <#if type.width == 4> >> public long getTwoAsLong(int index) { >> return data.getLong(index * ${type.width}); >> } >> >> >> >> Sorry I couldn't provide too much useful feedback without digging into the >> template, but this is any general feeling about these templates - too many >> "if" to types like "Interval" "Decimal" "Timestamp" >> >> >> >> On Tue, Oct 3, 2017 at 3:59 PM, Siddharth Teotia >> wrote: >> >> > I am in the middle of a simple prototype that has the basic >> implementation >> > of BaseFixedWidthVector, FixedValueVectorsPrototype.java (template) to >> > generate a simple IntVector using the proposal mentioned in the >> document. >> > >> > I have realized that even though the LOCs in existing templates are >> reduced >> > by 30-40% since bunch of common/basic functionality is moved to super >> class >> > BaseFixedWidthVector, the major source of pain (giant and complex if >> > conditions) associated with code generation is in accessor and mutator >> > which is still part of templates. >> > >> > I am trying to err on the side of not using templates at all since I >> feel >> > there is not much of gain from this refactoring project if the code in >> > templates is still complex and requires regular addition/modification >> when >> > adding new types. We are probably better off writing multiple sub >> classes >> > (with duplicate code as applicable) >> > >> > Thoughts? >> > >> > I can create a PR from this prototype code once it in reasonable shape >> for >> > review but was wondering if people have any opinion. >> > >> > Thanks, >> > Sidd >> > >> > On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotia >> > wrote: >> > >> > > Hi All, >> > > >> > > You should have received an invitation to edit the following document. >> > > Please feel free to add comments or additional content. >> > > >> > > https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930 >> > > sBKKaiyggOY7t4/edit >> > > >> > > Let me know if the document is not editable. >> > > >> > > Thanks, >> > > Siddharth >> > > >> > >> > >
Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.
Li, This is exactly what I was referring to in my previous email. I think if we have the opportunity (which we have now), we should see if complex template code like this can be removed unless it is absolutely necessary. This is why I was suggesting if we can just not have any code generated classes. Thanks, Siddharth On Tue, Oct 3, 2017 at 3:03 PM, Li Jinwrote: > Siddharth, > > Thanks for the update. Without really sit down and do the prototype, my > opinions can be wrong. But, > > I think a lot of the complication are code like this: > > <#elseif minor.class == "Decimal"> > public void get(int index, ${minor.class}Holder holder) { > holder.start = index * ${type.width}; > holder.buffer = data; > holder.scale = scale; > holder.precision = precision; > } > > The fact that we have to have a special block for Decimal is eyesore to me. > And it's not really shared with any other classes. > > Things like these are not great but might be ok? > > <#if type.width == 4> > public long getTwoAsLong(int index) { > return data.getLong(index * ${type.width}); > } > > > > Sorry I couldn't provide too much useful feedback without digging into the > template, but this is any general feeling about these templates - too many > "if" to types like "Interval" "Decimal" "Timestamp" > > > > On Tue, Oct 3, 2017 at 3:59 PM, Siddharth Teotia > wrote: > > > I am in the middle of a simple prototype that has the basic > implementation > > of BaseFixedWidthVector, FixedValueVectorsPrototype.java (template) to > > generate a simple IntVector using the proposal mentioned in the document. > > > > I have realized that even though the LOCs in existing templates are > reduced > > by 30-40% since bunch of common/basic functionality is moved to super > class > > BaseFixedWidthVector, the major source of pain (giant and complex if > > conditions) associated with code generation is in accessor and mutator > > which is still part of templates. > > > > I am trying to err on the side of not using templates at all since I feel > > there is not much of gain from this refactoring project if the code in > > templates is still complex and requires regular addition/modification > when > > adding new types. We are probably better off writing multiple sub classes > > (with duplicate code as applicable) > > > > Thoughts? > > > > I can create a PR from this prototype code once it in reasonable shape > for > > review but was wondering if people have any opinion. > > > > Thanks, > > Sidd > > > > On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotia > > wrote: > > > > > Hi All, > > > > > > You should have received an invitation to edit the following document. > > > Please feel free to add comments or additional content. > > > > > > https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930 > > > sBKKaiyggOY7t4/edit > > > > > > Let me know if the document is not editable. > > > > > > Thanks, > > > Siddharth > > > > > >
Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.
Siddharth, Thanks for the update. Without really sit down and do the prototype, my opinions can be wrong. But, I think a lot of the complication are code like this: <#elseif minor.class == "Decimal"> public void get(int index, ${minor.class}Holder holder) { holder.start = index * ${type.width}; holder.buffer = data; holder.scale = scale; holder.precision = precision; } The fact that we have to have a special block for Decimal is eyesore to me. And it's not really shared with any other classes. Things like these are not great but might be ok? <#if type.width == 4> public long getTwoAsLong(int index) { return data.getLong(index * ${type.width}); } Sorry I couldn't provide too much useful feedback without digging into the template, but this is any general feeling about these templates - too many "if" to types like "Interval" "Decimal" "Timestamp" On Tue, Oct 3, 2017 at 3:59 PM, Siddharth Teotiawrote: > I am in the middle of a simple prototype that has the basic implementation > of BaseFixedWidthVector, FixedValueVectorsPrototype.java (template) to > generate a simple IntVector using the proposal mentioned in the document. > > I have realized that even though the LOCs in existing templates are reduced > by 30-40% since bunch of common/basic functionality is moved to super class > BaseFixedWidthVector, the major source of pain (giant and complex if > conditions) associated with code generation is in accessor and mutator > which is still part of templates. > > I am trying to err on the side of not using templates at all since I feel > there is not much of gain from this refactoring project if the code in > templates is still complex and requires regular addition/modification when > adding new types. We are probably better off writing multiple sub classes > (with duplicate code as applicable) > > Thoughts? > > I can create a PR from this prototype code once it in reasonable shape for > review but was wondering if people have any opinion. > > Thanks, > Sidd > > On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotia > wrote: > > > Hi All, > > > > You should have received an invitation to edit the following document. > > Please feel free to add comments or additional content. > > > > https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930 > > sBKKaiyggOY7t4/edit > > > > Let me know if the document is not editable. > > > > Thanks, > > Siddharth > > >
[jira] [Created] (ARROW-1638) [Java] IPC roundtrip for null type
Wes McKinney created ARROW-1638: --- Summary: [Java] IPC roundtrip for null type Key: ARROW-1638 URL: https://issues.apache.org/jira/browse/ARROW-1638 Project: Apache Arrow Issue Type: New Feature Components: Java - Vectors Reporter: Wes McKinney Fix For: 0.8.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ARROW-1637) [C++] IPC round-trip for null type
Wes McKinney created ARROW-1637: --- Summary: [C++] IPC round-trip for null type Key: ARROW-1637 URL: https://issues.apache.org/jira/browse/ARROW-1637 Project: Apache Arrow Issue Type: New Feature Components: C++ Reporter: Wes McKinney Assignee: Wes McKinney Fix For: 0.8.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ARROW-1636) Integration tests for null type
Wes McKinney created ARROW-1636: --- Summary: Integration tests for null type Key: ARROW-1636 URL: https://issues.apache.org/jira/browse/ARROW-1636 Project: Apache Arrow Issue Type: New Feature Components: C++, Java - Vectors Reporter: Wes McKinney This was not implemented on the C++ side, and came up in ARROW-1584. Realistically arrays may be of null type, and we should be able to message these correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.
I am in the middle of a simple prototype that has the basic implementation of BaseFixedWidthVector, FixedValueVectorsPrototype.java (template) to generate a simple IntVector using the proposal mentioned in the document. I have realized that even though the LOCs in existing templates are reduced by 30-40% since bunch of common/basic functionality is moved to super class BaseFixedWidthVector, the major source of pain (giant and complex if conditions) associated with code generation is in accessor and mutator which is still part of templates. I am trying to err on the side of not using templates at all since I feel there is not much of gain from this refactoring project if the code in templates is still complex and requires regular addition/modification when adding new types. We are probably better off writing multiple sub classes (with duplicate code as applicable) Thoughts? I can create a PR from this prototype code once it in reasonable shape for review but was wondering if people have any opinion. Thanks, Sidd On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotiawrote: > Hi All, > > You should have received an invitation to edit the following document. > Please feel free to add comments or additional content. > > https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930 > sBKKaiyggOY7t4/edit > > Let me know if the document is not editable. > > Thanks, > Siddharth >
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Congrats Philip and Bryan :) On Tue, Oct 3, 2017 at 11:57 AM, Holden Karauwrote: > Congrats to the both of you :) Really excited to see the Areow project > continue to grow :) > > On Tue, Oct 3, 2017 at 10:24 AM Julian Hyde wrote: > > > Congratulations and welcome, Philip and Bryan! > > > > > On Oct 3, 2017, at 5:27 AM, Wes McKinney wrote: > > > > > > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > > > and Bryan Cutler have been invited to be Arrow committers. > > > > > > We are grateful for your contributions to the project and look forward > > > to growing the community together. > > > > > > Welcome, Phillip and Bryan, and congrats! > > > > > > - Wes > > > > -- > Twitter: https://twitter.com/holdenkarau >
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Congrats to the both of you :) Really excited to see the Areow project continue to grow :) On Tue, Oct 3, 2017 at 10:24 AM Julian Hydewrote: > Congratulations and welcome, Philip and Bryan! > > > On Oct 3, 2017, at 5:27 AM, Wes McKinney wrote: > > > > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > > and Bryan Cutler have been invited to be Arrow committers. > > > > We are grateful for your contributions to the project and look forward > > to growing the community together. > > > > Welcome, Phillip and Bryan, and congrats! > > > > - Wes > > -- Twitter: https://twitter.com/holdenkarau
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Congratulations and welcome, Philip and Bryan! > On Oct 3, 2017, at 5:27 AM, Wes McKinneywrote: > > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > and Bryan Cutler have been invited to be Arrow committers. > > We are grateful for your contributions to the project and look forward > to growing the community together. > > Welcome, Phillip and Bryan, and congrats! > > - Wes
Arrow sync call tomorrow 4 October @ 16:00 UTC
hi folks, We will be having our biweekly call again tomorrow. All are welcome to join; meeting notes will be published on the mailing list for further discussion and transparency. Can someone assist me with setting up a Google Meet call so that we can accommodate more than 10 attendees (we maxed out on a normal Google Hangout last time)? Thank you Wes
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
hi Srinivas, I've added you to the "Contributors" role in JIRA, so you can assign yourself JIRAs. Thanks On Tue, Oct 3, 2017 at 10:11 AM, Srinivas Reddywrote: > Hi, > > I'm Srinivas works at Kogentix and I would like to contribute to Arrow > project. > > Could you add me to contributors list? > > my JIRA username is: mrsrinivas > > > -Srinivas > > > > On 3 October 2017 at 17:57, Wes McKinney wrote: > >> On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud >> and Bryan Cutler have been invited to be Arrow committers. >> >> We are grateful for your contributions to the project and look forward >> to growing the community together. >> >> Welcome, Phillip and Bryan, and congrats! >> >> - Wes >>
[ANNOUNCE] Apache Arrow 0.7.1 released
The Apache Arrow community is pleased to announce the 0.7.1 release. This is a bug fix release from 0.7.0 and recommended for all users. It includes 25 resolved issues ([1]) since the 0.7.0 release. The release is available now from our website and [2]: http://arrow.apache.org/install/ Changelog http://arrow.apache.org/release/0.7.1.html Binary artifacts for respective languages should become available over the next day or so if they are not already. What is Apache Arrow? - Apache Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides low-overhead streaming and batch messaging, zero-copy interprocess communication (IPC), and vectorized in-memory analytics libraries. Please report any feedback to the mailing lists ([3]) Regards, The Apache Arrow community [1]: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20fixVersion%20%3D%200.7.1%20ORDER%20BY%20priority%20DESC [2]: https://www.apache.org/dyn/closer.cgi/arrow/arrow-0.7.1/ [3]: https://lists.apache.org/list.html?dev@arrow.apache.org
[jira] [Created] (ARROW-1635) Add release management guide for PMCs
Wes McKinney created ARROW-1635: --- Summary: Add release management guide for PMCs Key: ARROW-1635 URL: https://issues.apache.org/jira/browse/ARROW-1635 Project: Apache Arrow Issue Type: Improvement Reporter: Wes McKinney Assignee: Wes McKinney Fix For: 0.8.0 The Arrow release process has grown increasingly complex and time consuming, and there are many manual steps that a release manager must undertake before, during, and after a release vote. I will write a document to list all of these. Some of them desperately need to be more automated (like updating the online API documentation and website). Binary packaging is still not as automated as it could be. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Excited to be a committer! Looking forward to growing the project and community with everyone. Thanks! On Tue, Oct 3, 2017, 08:50 Li Jinwrote: > Congrats! > On Tue, Oct 3, 2017 at 8:27 AM Wes McKinney wrote: > > > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > > and Bryan Cutler have been invited to be Arrow committers. > > > > We are grateful for your contributions to the project and look forward > > to growing the community together. > > > > Welcome, Phillip and Bryan, and congrats! > > > > - Wes > > >
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Hi, I'm Srinivas works at Kogentix and I would like to contribute to Arrow project. Could you add me to contributors list? my JIRA username is: mrsrinivas -Srinivas On 3 October 2017 at 17:57, Wes McKinneywrote: > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > and Bryan Cutler have been invited to be Arrow committers. > > We are grateful for your contributions to the project and look forward > to growing the community together. > > Welcome, Phillip and Bryan, and congrats! > > - Wes >
Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
Congrats! On Tue, Oct 3, 2017 at 8:27 AM Wes McKinneywrote: > On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > and Bryan Cutler have been invited to be Arrow committers. > > We are grateful for your contributions to the project and look forward > to growing the community together. > > Welcome, Phillip and Bryan, and congrats! > > - Wes >
[ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler
On behalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud and Bryan Cutler have been invited to be Arrow committers. We are grateful for your contributions to the project and look forward to growing the community together. Welcome, Phillip and Bryan, and congrats! - Wes