Re: First impressions with Drill+Parquet+S3

2016-10-28 Thread Uwe Korn
00:26, Parth Chandra wrote: Hi Uwe, Can you log JIRA's for the performance issues that you encounter while working on S3? Not many folks are working on optimizing that path, so any patches that you might be able to contribute would be appreciated. Parth On Thu, Oct 6, 2016 at 1:56 PM, Uwe Korn

Re: First impressions with Drill+Parquet+S3

2016-10-06 Thread Uwe Korn
that are not touched often and when, only a few queries are done on them. > Am 06.10.2016 um 22:47 schrieb Ted Dunning <ted.dunn...@gmail.com>: > > Have you tried running against a real file system interface? Or even just > against HDFS? > > > > On Thu, Oct 6,

First impressions with Drill+Parquet+S3

2016-10-06 Thread Uwe Korn
Hello, We had some test runs with Drill 1.8 in the last days and wanted to share the experience with you as we've made some interesting findings that astonished us. We did run on our internal company cluster and thus used the S3 API to access our internal storage cluster, not AWS (the behavior

Re: S3 Storage Plugin not working for Signature V4 regions

2016-10-04 Thread Uwe Korn
Hello, this could probably be that problem that the AWS SDK coming with the drill version is quite old. I’m doing some tests currently with Drill 1.8 + a custom Hadoop 3.0.0-alpha1 version that was patched to use the latest AWS SDK. That should not only give you access to V4 regions but it

Re: Cannot load Parquet files created with parquet-cpp in Drill

2016-09-07 Thread Uwe Korn
can read that. Regards Kunal On Wed 7-Sep-2016 5:30:32 AM, Uwe Korn <uw...@xhochy.com> wrote: Hello, I'm currently looking at the correctness of our C++ implementation of Parquet and noticed that I cannot load these files in Drill. Although this is probably a bug in the C++ implementat

Cannot load Parquet files created with parquet-cpp in Drill

2016-09-07 Thread Uwe Korn
Hello, I'm currently looking at the correctness of our C++ implementation of Parquet and noticed that I cannot load these files in Drill. Although this is probably a bug in the C++ implementation, I don't understand what causes the error. Using the Java parquet-tools, I can read these files.