Re: [DISCUSS] [Rust] Adding support for Flight protocol
Hi Andy, I've fixed the issue and left a description of the problem on the PR. On Sun, 17 Nov 2019 at 19:23, Andy Grove wrote: > I'm now trying to create a Flight server in Rust and am struggling a bit. > See https://github.com/apache/arrow/pull/5852 for more information. If > anyone is available to take a look I'd appreciate it. > > I'm going to reach out to Lucio directly since he isn't currently > subscribed to the mailing list IIRC. > > Thanks, > > Andy. > > > > On Fri, Oct 18, 2019 at 11:17 AM Lucio Franco > wrote: > > > Hi all! > > > > I am the author of Tonic, I'd love to see the rust flight implementation > > done with Tonic. David, it looks like what you implemented with > tower-grpc > > should work just fine with tonic as well. I am also interested in helping > > out. Since, I guess most of my experience up to now is with Tonic > itself, I > > wanted to make myself available to you all for help or special > > implementations coming from the Tonic side. I am also willing to help > with > > the flight rust implementation. So please let me know around that what I > > can do! > > Thanks, > > Lucio > > > > On Oct 17 2019, at 7:26 pm, Neville Dipale > wrote: > > > Thanks Andy, > > > > > > Please see > > https://github.com/apache/arrow/pull/4167#issuecomment-543381089 > > > for the status of the PR. We have a few missing data types (fixed list, > > > timezone to timestamp, etc.) that are currently stopping me from > testing > > > the reading of files. > > > > > > I'm trying out creating a fixed size list, and I'll open a PR for that > if > > > my attempt works. > > > > > > On Fri, 18 Oct 2019 at 01:10, Andy Grove > wrote: > > > > Thanks for all the updates. I'd like to get involved and help out > with > > this > > > > effort as well. I don't have any major work planned for DataFusion > for > > > > 1.0.0 now other than maybe moving to the new parquet ArrowReader, if > > it is > > > > ready in time. > > > > > > > > I have been chatting with the author of the Rust Flatbuffer project > > about > > > > some of the issues and I can take an action to follow up with that. > > > > > > > > I have also been talking with one of the authors of Tonic, and I > > believe > > > > they might be interested in helping here too. > > > > > > > > Let me know how else I can help out with this effort. > > > > Andy. > > > > > > > > > > > > On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale < > nevilled...@gmail.com> > > > > wrote: > > > > > > > > > Good evening > > > > > With support for testing against integration files now done, I've > > resumed > > > > > work on the IPC reader. If I don't encounter trouble reading the > > existing > > > > > files, I expect to be done with this work by the end of the > weekend. > > I > > > > > > > > had > > > > > taken the approach of one large PR to include all Rust-supported > > Arrow > > > > > types. > > > > > > > > > > I'm not sure of how long the writer would take, but I remain > > committed to > > > > > having this work completed by 1.0. > > > > > > > > > > I have to catch up on null-type roundtrip and the padding alignment > > work > > > > as > > > > > I haven't been able to keep abreast with development these last few > > > > > > > > months. > > > > > Also, the Rust flatbuffer issues that we've had haven't progressed > > in the > > > > > relevant repo, so it still makes ergonomics not great, but at least > > our > > > > > users don't have to worry about that. > > > > > > > > > > @Andy I have good experience with gRPC in Rust, and also want to > see > > > > Flight > > > > > support landing soon. > > > > > > > > > > On Thu, 17 Oct 2019, 17:06 David Li, > wrote: > > > > > > Just for reference, it's possible once the basic IPC support is > > > > merged; I > > > > > > had a proof of concept, though it needs to be updated to use > Tonic > > over > > > > > > tower-grpc, actually implement the zero-copy optimizations, > > provide a > > > > > > > > > > real > > > > > > API, etc. > > > > > > > > > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney > > wrote: > > > > > > > I hope to see Flight in all the reference implementations > > eventually. > > > > > > > Having hardened IPC support is a pre-requisite, it would be > > ideal to > > > > > > > have Rust as a participant in the integration tests > > > > > > > > > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove < > > andygrov...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > I was approached directly about adding Flight support to the > > Rust > > > > > > > > implementation, and said I would start a discussion here on > the > > > > > > > > > > > > > > > > > > > > > > > mailing > > > > > > > > list. > > > > > > > > > > > > > > > > There is ongoing work with IPC and integration and I believe > > that > > > > it > > > > > > > would > > > > > > > > make sense to start looking at adding Flight support. > > > > > > > > > > > > > > > > I'd like to hear what
Re: [DISCUSS] [Rust] Adding support for Flight protocol
I'm now trying to create a Flight server in Rust and am struggling a bit. See https://github.com/apache/arrow/pull/5852 for more information. If anyone is available to take a look I'd appreciate it. I'm going to reach out to Lucio directly since he isn't currently subscribed to the mailing list IIRC. Thanks, Andy. On Fri, Oct 18, 2019 at 11:17 AM Lucio Franco wrote: > Hi all! > > I am the author of Tonic, I'd love to see the rust flight implementation > done with Tonic. David, it looks like what you implemented with tower-grpc > should work just fine with tonic as well. I am also interested in helping > out. Since, I guess most of my experience up to now is with Tonic itself, I > wanted to make myself available to you all for help or special > implementations coming from the Tonic side. I am also willing to help with > the flight rust implementation. So please let me know around that what I > can do! > Thanks, > Lucio > > On Oct 17 2019, at 7:26 pm, Neville Dipale wrote: > > Thanks Andy, > > > > Please see > https://github.com/apache/arrow/pull/4167#issuecomment-543381089 > > for the status of the PR. We have a few missing data types (fixed list, > > timezone to timestamp, etc.) that are currently stopping me from testing > > the reading of files. > > > > I'm trying out creating a fixed size list, and I'll open a PR for that if > > my attempt works. > > > > On Fri, 18 Oct 2019 at 01:10, Andy Grove wrote: > > > Thanks for all the updates. I'd like to get involved and help out with > this > > > effort as well. I don't have any major work planned for DataFusion for > > > 1.0.0 now other than maybe moving to the new parquet ArrowReader, if > it is > > > ready in time. > > > > > > I have been chatting with the author of the Rust Flatbuffer project > about > > > some of the issues and I can take an action to follow up with that. > > > > > > I have also been talking with one of the authors of Tonic, and I > believe > > > they might be interested in helping here too. > > > > > > Let me know how else I can help out with this effort. > > > Andy. > > > > > > > > > On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale > > > wrote: > > > > > > > Good evening > > > > With support for testing against integration files now done, I've > resumed > > > > work on the IPC reader. If I don't encounter trouble reading the > existing > > > > files, I expect to be done with this work by the end of the weekend. > I > > > > > > had > > > > taken the approach of one large PR to include all Rust-supported > Arrow > > > > types. > > > > > > > > I'm not sure of how long the writer would take, but I remain > committed to > > > > having this work completed by 1.0. > > > > > > > > I have to catch up on null-type roundtrip and the padding alignment > work > > > as > > > > I haven't been able to keep abreast with development these last few > > > > > > months. > > > > Also, the Rust flatbuffer issues that we've had haven't progressed > in the > > > > relevant repo, so it still makes ergonomics not great, but at least > our > > > > users don't have to worry about that. > > > > > > > > @Andy I have good experience with gRPC in Rust, and also want to see > > > Flight > > > > support landing soon. > > > > > > > > On Thu, 17 Oct 2019, 17:06 David Li, wrote: > > > > > Just for reference, it's possible once the basic IPC support is > > > merged; I > > > > > had a proof of concept, though it needs to be updated to use Tonic > over > > > > > tower-grpc, actually implement the zero-copy optimizations, > provide a > > > > > > > > real > > > > > API, etc. > > > > > > > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney > wrote: > > > > > > I hope to see Flight in all the reference implementations > eventually. > > > > > > Having hardened IPC support is a pre-requisite, it would be > ideal to > > > > > > have Rust as a participant in the integration tests > > > > > > > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove < > andygrov...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > I was approached directly about adding Flight support to the > Rust > > > > > > > implementation, and said I would start a discussion here on the > > > > > > > > > > > > > > > > > > > mailing > > > > > > > list. > > > > > > > > > > > > > > There is ongoing work with IPC and integration and I believe > that > > > it > > > > > > would > > > > > > > make sense to start looking at adding Flight support. > > > > > > > > > > > > > > I'd like to hear what others think though. > > > > > > > Thanks, > > > > > > > Andy.
Re: [DISCUSS] [Rust] Adding support for Flight protocol
Hi all! I am the author of Tonic, I'd love to see the rust flight implementation done with Tonic. David, it looks like what you implemented with tower-grpc should work just fine with tonic as well. I am also interested in helping out. Since, I guess most of my experience up to now is with Tonic itself, I wanted to make myself available to you all for help or special implementations coming from the Tonic side. I am also willing to help with the flight rust implementation. So please let me know around that what I can do! Thanks, Lucio On Oct 17 2019, at 7:26 pm, Neville Dipale wrote: > Thanks Andy, > > Please see https://github.com/apache/arrow/pull/4167#issuecomment-543381089 > for the status of the PR. We have a few missing data types (fixed list, > timezone to timestamp, etc.) that are currently stopping me from testing > the reading of files. > > I'm trying out creating a fixed size list, and I'll open a PR for that if > my attempt works. > > On Fri, 18 Oct 2019 at 01:10, Andy Grove wrote: > > Thanks for all the updates. I'd like to get involved and help out with this > > effort as well. I don't have any major work planned for DataFusion for > > 1.0.0 now other than maybe moving to the new parquet ArrowReader, if it is > > ready in time. > > > > I have been chatting with the author of the Rust Flatbuffer project about > > some of the issues and I can take an action to follow up with that. > > > > I have also been talking with one of the authors of Tonic, and I believe > > they might be interested in helping here too. > > > > Let me know how else I can help out with this effort. > > Andy. > > > > > > On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale > > wrote: > > > > > Good evening > > > With support for testing against integration files now done, I've resumed > > > work on the IPC reader. If I don't encounter trouble reading the existing > > > files, I expect to be done with this work by the end of the weekend. I > > > > had > > > taken the approach of one large PR to include all Rust-supported Arrow > > > types. > > > > > > I'm not sure of how long the writer would take, but I remain committed to > > > having this work completed by 1.0. > > > > > > I have to catch up on null-type roundtrip and the padding alignment work > > as > > > I haven't been able to keep abreast with development these last few > > > > months. > > > Also, the Rust flatbuffer issues that we've had haven't progressed in the > > > relevant repo, so it still makes ergonomics not great, but at least our > > > users don't have to worry about that. > > > > > > @Andy I have good experience with gRPC in Rust, and also want to see > > Flight > > > support landing soon. > > > > > > On Thu, 17 Oct 2019, 17:06 David Li, wrote: > > > > Just for reference, it's possible once the basic IPC support is > > merged; I > > > > had a proof of concept, though it needs to be updated to use Tonic over > > > > tower-grpc, actually implement the zero-copy optimizations, provide a > > > > > > real > > > > API, etc. > > > > > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney wrote: > > > > > I hope to see Flight in all the reference implementations eventually. > > > > > Having hardened IPC support is a pre-requisite, it would be ideal to > > > > > have Rust as a participant in the integration tests > > > > > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove > > > > wrote: > > > > > > > > > > > > I was approached directly about adding Flight support to the Rust > > > > > > implementation, and said I would start a discussion here on the > > > > > > > > > > > > > > > mailing > > > > > > list. > > > > > > > > > > > > There is ongoing work with IPC and integration and I believe that > > it > > > > > would > > > > > > make sense to start looking at adding Flight support. > > > > > > > > > > > > I'd like to hear what others think though. > > > > > > Thanks, > > > > > > Andy.
Re: [DISCUSS] [Rust] Adding support for Flight protocol
Thanks Andy, Please see https://github.com/apache/arrow/pull/4167#issuecomment-543381089 for the status of the PR. We have a few missing data types (fixed list, timezone to timestamp, etc.) that are currently stopping me from testing the reading of files. I'm trying out creating a fixed size list, and I'll open a PR for that if my attempt works. On Fri, 18 Oct 2019 at 01:10, Andy Grove wrote: > Thanks for all the updates. I'd like to get involved and help out with this > effort as well. I don't have any major work planned for DataFusion for > 1.0.0 now other than maybe moving to the new parquet ArrowReader, if it is > ready in time. > > I have been chatting with the author of the Rust Flatbuffer project about > some of the issues and I can take an action to follow up with that. > > I have also been talking with one of the authors of Tonic, and I believe > they might be interested in helping here too. > > Let me know how else I can help out with this effort. > > Andy. > > > > On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale > wrote: > > > Good evening > > > > With support for testing against integration files now done, I've resumed > > work on the IPC reader. If I don't encounter trouble reading the existing > > files, I expect to be done with this work by the end of the weekend. I > had > > taken the approach of one large PR to include all Rust-supported Arrow > > types. > > > > I'm not sure of how long the writer would take, but I remain committed to > > having this work completed by 1.0. > > > > I have to catch up on null-type roundtrip and the padding alignment work > as > > I haven't been able to keep abreast with development these last few > months. > > Also, the Rust flatbuffer issues that we've had haven't progressed in the > > relevant repo, so it still makes ergonomics not great, but at least our > > users don't have to worry about that. > > > > @Andy I have good experience with gRPC in Rust, and also want to see > Flight > > support landing soon. > > > > On Thu, 17 Oct 2019, 17:06 David Li, wrote: > > > > > Just for reference, it's possible once the basic IPC support is > merged; I > > > had a proof of concept, though it needs to be updated to use Tonic over > > > tower-grpc, actually implement the zero-copy optimizations, provide a > > real > > > API, etc. > > > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney wrote: > > > > > > > I hope to see Flight in all the reference implementations eventually. > > > > > > > > Having hardened IPC support is a pre-requisite, it would be ideal to > > > > have Rust as a participant in the integration tests > > > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove > > > wrote: > > > > > > > > > > I was approached directly about adding Flight support to the Rust > > > > > implementation, and said I would start a discussion here on the > > mailing > > > > > list. > > > > > > > > > > There is ongoing work with IPC and integration and I believe that > it > > > > would > > > > > make sense to start looking at adding Flight support. > > > > > > > > > > I'd like to hear what others think though. > > > > > > > > > > Thanks, > > > > > > > > > > Andy. > > > > > > > > > >
Re: [DISCUSS] [Rust] Adding support for Flight protocol
Thanks for all the updates. I'd like to get involved and help out with this effort as well. I don't have any major work planned for DataFusion for 1.0.0 now other than maybe moving to the new parquet ArrowReader, if it is ready in time. I have been chatting with the author of the Rust Flatbuffer project about some of the issues and I can take an action to follow up with that. I have also been talking with one of the authors of Tonic, and I believe they might be interested in helping here too. Let me know how else I can help out with this effort. Andy. On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale wrote: > Good evening > > With support for testing against integration files now done, I've resumed > work on the IPC reader. If I don't encounter trouble reading the existing > files, I expect to be done with this work by the end of the weekend. I had > taken the approach of one large PR to include all Rust-supported Arrow > types. > > I'm not sure of how long the writer would take, but I remain committed to > having this work completed by 1.0. > > I have to catch up on null-type roundtrip and the padding alignment work as > I haven't been able to keep abreast with development these last few months. > Also, the Rust flatbuffer issues that we've had haven't progressed in the > relevant repo, so it still makes ergonomics not great, but at least our > users don't have to worry about that. > > @Andy I have good experience with gRPC in Rust, and also want to see Flight > support landing soon. > > On Thu, 17 Oct 2019, 17:06 David Li, wrote: > > > Just for reference, it's possible once the basic IPC support is merged; I > > had a proof of concept, though it needs to be updated to use Tonic over > > tower-grpc, actually implement the zero-copy optimizations, provide a > real > > API, etc. > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney wrote: > > > > > I hope to see Flight in all the reference implementations eventually. > > > > > > Having hardened IPC support is a pre-requisite, it would be ideal to > > > have Rust as a participant in the integration tests > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove > > wrote: > > > > > > > > I was approached directly about adding Flight support to the Rust > > > > implementation, and said I would start a discussion here on the > mailing > > > > list. > > > > > > > > There is ongoing work with IPC and integration and I believe that it > > > would > > > > make sense to start looking at adding Flight support. > > > > > > > > I'd like to hear what others think though. > > > > > > > > Thanks, > > > > > > > > Andy. > > > > > >
Re: [DISCUSS] [Rust] Adding support for Flight protocol
Just for reference, it's possible once the basic IPC support is merged; I had a proof of concept, though it needs to be updated to use Tonic over tower-grpc, actually implement the zero-copy optimizations, provide a real API, etc. https://github.com/apache/arrow/pull/4167#issuecomment-529695811 On Thu, Oct 17, 2019, 10:51 Wes McKinney wrote: > I hope to see Flight in all the reference implementations eventually. > > Having hardened IPC support is a pre-requisite, it would be ideal to > have Rust as a participant in the integration tests > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove wrote: > > > > I was approached directly about adding Flight support to the Rust > > implementation, and said I would start a discussion here on the mailing > > list. > > > > There is ongoing work with IPC and integration and I believe that it > would > > make sense to start looking at adding Flight support. > > > > I'd like to hear what others think though. > > > > Thanks, > > > > Andy. >
Re: [DISCUSS] [Rust] Adding support for Flight protocol
I hope to see Flight in all the reference implementations eventually. Having hardened IPC support is a pre-requisite, it would be ideal to have Rust as a participant in the integration tests On Thu, Oct 17, 2019 at 9:41 AM Andy Grove wrote: > > I was approached directly about adding Flight support to the Rust > implementation, and said I would start a discussion here on the mailing > list. > > There is ongoing work with IPC and integration and I believe that it would > make sense to start looking at adding Flight support. > > I'd like to hear what others think though. > > Thanks, > > Andy.
