[jira] [Created] (DRILL-5918) Allow manual configuration when using Runtime.getRuntime().availableProcessors()
Elijah Zupancic created DRILL-5918: -- Summary: Allow manual configuration when using Runtime.getRuntime().availableProcessors() Key: DRILL-5918 URL: https://issues.apache.org/jira/browse/DRILL-5918 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.11.0 Reporter: Elijah Zupancic Priority: Critical Currently Drill auto-configures the number of threads in various thread pools based on the processor count: https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java#L110 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java#L269 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/AbstractResourceManager.java#L55 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java#L99 In a number of situations, this configuration is incorrect. In particular, the settings aren't correct when [running in a bare-metal container](https://docs.google.com/document/d/1WGPGiJtbJZPZBMSD9YHAAz-O4AlAET9qOnS_GeriZtE/edit?usp=sharing) because you don't necessarily have access to all of the compute shares or memory of the underlying host. Ideally, the number of CPUs to use could be specified as a system parameter to Drill. This would allow users to configure Drill to their optimal settings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Manta Object Store Support
Please disregard this. I’ve figured out how to get it working. Once I realized that error logging for Hadoop Filesystem drivers wasn’t enabled by logback by default, then everything clicked into place. I’ll be back to ask more questions. Thank you for your time, Elijah Zupancic > On May 6, 2016, at 9:43 AM, Elijah Zupancic wrote: > > I'm trying to get started contributing to Apache Drill. I've got the project > checked out and it is building to my satisfaction. Right now, I'm trying to > add support for the open source object store Manta > (https://github.com/joyent/manta <https://github.com/joyent/manta>). I > thought that this would be a good learning project. > > Initially, I want to add support in the same way that S3 has support. > However, I can't seem to find a reference to the S3 storage driver in the > code base. Is the s3 storage driver part of a different project? How would > you suggest that I get started? > > Thank you, > Elijah Zupancic
Re: Manta Object Store Support
Hi Tomer, Thanks for your advice about creating a Hadoop FileSystem implementation. I just finished a prototypical implementation of a Hadoop file system for Manta: https://github.com/dekobon/hadoop-manta <https://github.com/dekobon/hadoop-manta> I see the example for enabling S3 with Apache Drill and I’ve verified that it works. However, when I attempt to replicate the configuration of S3 for Manta, I’m unable to get the Hadoop FileSystem driver to load. I’ve verified that the FileSystem driver works in Hadoop by checking all of the hdfs dfs -* commands and I’ve got a fair bit of automated testing around it. What’s the magic to get it turned on with Drill? Do I need to do something to make the jar load other than copy it into jars/3rdparty? Right now, I’m just testing in drill-embedded for what it is worth. Thanks, Elijah Zupancic > On May 6, 2016, at 4:11 PM, Tomer Shiran wrote: > > Does Manta have a Hadoop FileSystem API implementation? That's what Drill > uses for S3, HDFS, MapR-FS, Azure Blob Storage, etc. You could potentially > write a Drill storage plugin, but you get a lot for free if you already > have the file system implementation. > > On Fri, May 6, 2016 at 9:43 AM, Elijah Zupancic wrote: > >> I'm trying to get started contributing to Apache Drill. I've got the >> project checked out and it is building to my satisfaction. Right now, I'm >> trying to add support for the open source object store Manta ( >> https://github.com/joyent/manta). I thought that this would be a good >> learning project. >> >> Initially, I want to add support in the same way that S3 has support. >> However, I can't seem to find a reference to the S3 storage driver in the >> code base. Is the s3 storage driver part of a different project? How would >> you suggest that I get started? >> >> Thank you, >> Elijah Zupancic >>
Getting Started & Adding Manta Object Store Support
I'm trying to get started contributing to Apache Drill. I've got the project checked out and it is building to my satisfaction. Right now, I'm trying to add support for the open source object store Manta (https://github.com/joyent/manta <https://github.com/joyent/manta>). I thought that this would be a good learning project. Initially, I want to add support in the same way that S3 has support. However, I can't seem to find a reference to the S3 storage driver in the code base. Is the s3 storage driver part of a different project? How would you suggest that I get started? Thank you, Elijah Zupancic
Manta Object Store Support
I'm trying to get started contributing to Apache Drill. I've got the project checked out and it is building to my satisfaction. Right now, I'm trying to add support for the open source object store Manta ( https://github.com/joyent/manta). I thought that this would be a good learning project. Initially, I want to add support in the same way that S3 has support. However, I can't seem to find a reference to the S3 storage driver in the code base. Is the s3 storage driver part of a different project? How would you suggest that I get started? Thank you, Elijah Zupancic
