tmp noexec

2016-07-25 Thread scott
dLinkError: /tmp/libnetty-transport-native-epoll5743269078378802025.so: /tmp/libnetty-transport-native-epoll5743269078378802025.so: failed to map segment from shared object: Operation not permitted Does anyone know of a way to configure Drill to use a different tmp location? Thanks, Scott

Re: tmp noexec

2016-07-26 Thread scott
Thanks Leon for the suggestion, but do you think this config change will help with my startup problem? It looks like it changes operations for sort after startup. Scott On Mon, Jul 25, 2016 at 3:55 PM, Leon Clayton <lclay...@maprtech.com> wrote: > > I move the /tmp of

Re: Drill Performance

2016-07-14 Thread scott
Curious what the biggest is. Has anyone configured more than 100 drillbits in a cluster before? Scott On 07/14/2016 10:27 AM, Ted Dunning wrote: On the right distribution, you can restrict the subset of the cluster that has the data you need to avoid locality variation when Drill only runs

Re: Drill Performance

2016-07-07 Thread scott
Follow up question: Is there a sweet spot for DRILL_MAX_DIRECT_MEMORY and DRILL_HEAP settings? On Wed, Jul 6, 2016 at 2:42 PM, scott <tcots8...@gmail.com> wrote: > Hello, > Does anyone know if there is a maximum number of drillbits recommended in > a Drill cluster? For example

Re: Drill Performance

2016-07-07 Thread scott
resulting in a plateau or decline of performance? Thanks, Scott On Thu, Jul 7, 2016 at 5:00 PM, Abdel Hakim Deneche <adene...@maprtech.com> wrote: > I'm not sure you'll get any performance improvement from running more than > a single drillbit per cluster node. > > On Thu, Jul

Drill Performance

2016-07-06 Thread scott
together? Thanks, Scott

Re: tmp noexec

2016-07-29 Thread scott
Thanks Jacques. I think I have solved my immediate problem. I added -Djava.io.tmpdir=/newtmp to the DRILL_JAVA_OPTS variable in drill-env.sh. I am now able to start drillbit without error. Scott On Fri, Jul 29, 2016 at 4:29 PM, Jacques Nadeau <jacq...@dremio.com> wrote: > Unfo

User Impersonation

2016-06-30 Thread scott
I misunderstanding how user impersonation works? Thanks for your time, Scott

Re: User Impersonation

2016-06-30 Thread scott
Impersonation using the default dfs configuration is not supported? The documentation for Impersonation Support says that File System is a supported Storage Plugin, and that only HBase is not supported. If this is true, do you know if there is a Jira ticket to add this feature? Scott On Thu, Jun

Re: User Impersonation

2016-07-01 Thread scott
Am I able to open a Jira ticket on this? Or, is this something a developer has to do? Scott On Thu, Jun 30, 2016 at 5:17 PM, scott <tcots8...@gmail.com> wrote: > Impersonation using the default dfs configuration is not supported? The > documentation for Impersonation Support sa

Re: User Impersonation

2016-07-01 Thread scott
Great explanation, Ted. I think I will still open the ticket, if nothing else to address the gaps in documentation. Thanks, Scott On Fri, Jul 1, 2016 at 2:30 PM, Ted Dunning <ted.dunn...@gmail.com> wrote: > You can certainly open a ticket for this. > > The problem is resol

Re: user impersonation bypass

2016-08-16 Thread scott
Chun, You were right. Thanks for pointing me in the right direction. One of my drillbits didn't get my configuration change, and by chance the zookeeper connection didn't assign it to me. I'll keep a close eye out for this situation in the future. Scott On Mon, Aug 15, 2016 at 9:38 PM, Chun

user impersonation bypass

2016-08-15 Thread scott
of the drillbits using port 31010 with same login credentials. This time, I had full access to everything in maprfs, as if I were logged in as the mapr user. Has anyone else experienced this or has an explanation for it? Thanks, Scott

Re: Drill Queries Timing Out

2016-08-31 Thread scott
I found the issue. Drill was fine, but I had one of the new nodes which had IP tables blocking some ports. Good lesson learned here for me. Thanks, Scott On Wed, Aug 31, 2016 at 3:55 PM, Jinfeng Ni <j...@apache.org> wrote: > Can you please check the query profiles for both cases

Drill Queries Timing Out

2016-08-31 Thread scott
tell me if there are some configuration settings that can improve this? Or, is there some point where Drill performance decreases when the size of the cluster is too large? Thanks, Scott

Connect multiple drill clusters

2016-08-24 Thread scott
Hello, I'm wondering if it is possible to connect two Drill clusters? For instance, could I add a JDBC storage plugin in one cluster that connects to another Drill cluster? Thanks, Scott

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread scott
hat do you think? On Mon, Aug 27, 2018 at 11:58 PM Paul Rogers wrote: > Hi Scott, > > I created a file, "test.json", using the data from your e-mail: > > [ { "var1": "foo", "var2":"bar"},{"var1": "fo", "

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread scott
Paul, Thank you for the workaround, that worked in my case perfectly !! Scott On Tue, Aug 28, 2018 at 12:20 PM Lee, David wrote: > This is a pretty ugly json file.. 568 megs for 7227 records.. > > => ls -l test.jsonl > -rw-r--r-- 1 my_login users 568693075 Aug 28 1

Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-27 Thread scott
lution since this was reported 4 years back. Does anyone have a solution or workaround to this problem? Thanks, Scott

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-27 Thread scott
Paul, I'm using version 1.12. Can you tell me what version you think that was fixed in? The ticket I referenced is still open, with no comments. Scott On Mon, Aug 27, 2018 at 5:47 PM Paul Rogers wrote: > Hi David, > > JSON files are never splittable: there is no single-character wa

Re: query performance with unequal drillbits

2018-08-22 Thread scott
the weaker ones. > > This increases overheads, but it might help you out. > > > > On Tue, Aug 21, 2018 at 1:48 PM scott wrote: > > > Hi community, > > I am trying to find a way to tune Drill so that weaker drillbits get less > > data to work on so that the

CTAS memory leak

2018-08-29 Thread scott
: -Xms8G -Xmx8G -XX:MaxDirectMemorySize=10G on a server that has 48G RAM. Has anyone else experienced this? Are there any workarounds you can suggest? Thanks for your time, Scott

Re: Drill on Windows

2018-04-19 Thread scott
appreciated if you built one. Scott On 04/19/2018 11:21 AM, Timothy Farkas wrote: Rahul / Chris, The easiest way to do this would be to create a docker container for Drill. Also it looks like Docker leverage's Windows 10's linux subsystem so docker images don't run in a VM https://www.hanselman.com

Re: Drill Specific Performance Monitoring Utilities

2015-01-17 Thread Jim Scott
http://mufeed.livejournal.com -- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com

Re: Connection timeout 1.0.0

2015-05-20 Thread Jim Scott
jdbc:drill:zk=127.0.0.1:31010` without success. Thanks Davide -- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training

drill configuration setting - rows overwriting one another

2015-06-25 Thread Jim Scott
Can anyone point me to a configuration setting (not maxwidth) that can prevent having the drill output in the CLI from overwriting data that is already in a row? (i.e. the data is not wrapping to a new line) I have a query that when I select all the columns, the data in the columns at the end

Re: Drill Installation Question

2015-08-17 Thread Jim Scott
. Please let me know, any info on the above points would be helpful. Thanks, Naveen -- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 @kingmesal https://twitter.com/kingmesal http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com Now Available - Free

Re: Will drill work with Java SE 8 versions?

2015-08-20 Thread Jim Scott
work on the Java SE JDK 8 versions? Thanks, Joe -- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 @kingmesal https://twitter.com/kingmesal http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com Now Available - Free Hadoop On-Demand

Re: drill configuration setting - rows overwriting one another

2015-06-26 Thread Jim Scott
to work around this. On Jun 25, 2015 1:13 PM, Jim Scott jsc...@maprtech.com wrote: Can anyone point me to a configuration setting (not maxwidth) that can prevent having the drill output in the CLI from overwriting data that is already in a row? (i.e. the data is not wrapping to a new line) I

Re: IPv6 in Drill/Parquet

2015-07-24 Thread Jim Scott
, 2015 at 2:39 PM, Jim Scott jsc...@maprtech.com wrote: an IPv6 address is actually two longs. Depending on the type of analysis you are doing you may prefer to store them that way. e.g. the range on the left side is a home / location and the range on the right side are sub values

Re: geeting schema from a csv file in json response instead of columns array

2015-07-26 Thread Jim Scott
That functionality does not yet exist. There is a JIRA open for that functionality to be added. But has not yet been prioritized high enough to get completed. Is this something you are interested in contributing toward? In the mean time, your options are: * Alias (select columns[0] as field

Re: IPv6 in Drill/Parquet

2015-07-24 Thread Jim Scott
/asdrill) now also includes a IPv4 conversion (One may exist already) . Regards, -Stefan -- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com Now Available - Free Hadoop On-Demand Training http

Re: Drill on Mesos - A Story

2015-07-18 Thread Jim Scott
-- *Jim Scott* Director, Enterprise Strategy Architecture +1 (347) 746-9281 http://www.mapr.com/ [image: MapR Technologies] http://www.mapr.com Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available

Re: Httpd logs with Apach Drill 1.2

2015-10-27 Thread Jim Scott
el > like there are also other things pressing so I probably won't come back to > it for a bit. If someone else has the time to move it forward, that would > be great. > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Sun, Oct 18, 2015 at 3:21 PM, Jim Scott <jsc...@maprt

Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

2015-11-17 Thread scott cote
the aws command line tool for sanity check. SCott > On Nov 17, 2015, at 1:36 AM, Andries Engelbrecht <aengelbre...@maprtech.com> > wrote: > > For large cluster I normally use clush (cluster shell), edit the config files > on one node and then copy them to all using clu

Re: Apache Drill connecting to OpenTSDB hbase

2015-10-20 Thread Jim Scott
arate window and never comes back. > >>>> > >>>> > >>>> > >>>> I have tried to configure the settings in Drill webpage - Storage > with the following settings: > >>>> > >>>> > >>>> > >>

Re: S3 data source - invalid URI

2015-10-15 Thread scott cote
I have had problems with hyphens in the s3 names in the past when used with drill. try removing the hyphen. SCott > On Oct 15, 2015, at 11:03 AM, Jason Jho <jason...@gmail.com> wrote: > > I've enabled JetS3t and configured a storage plugin for S3, but I'm having > trouble c

Re: xml files with Drill

2015-09-04 Thread Jim Scott
ich...@lancetdatasciences.com> wrote: > Looking for documentation, example or syntax to read xml files with Drill. > > Is there a way to invoke an xml parser in Drill? > > Thanks- > hrich...@lancetdatasciences.com > > -- *Jim Scott* Director, Enterprise Strategy & Archite

Re: CSV Bug

2015-09-17 Thread Jim Scott
tc", "USER AGENT"] > > I'm having this response (3columns): > ["XXX.XXX.XXX.XXX", "200", "GET / ... etc\" \"USER AGENT\""] > > But if I edit the file with comma delimiter a the configuration, it's > working fine. > I

Re: Protobuf Files

2015-10-03 Thread Jim Scott
somebody else have a good idea for this? > > > On Fri, Oct 2, 2015 at 7:17 AM, Jim Scott <jsc...@maprtech.com> wrote: > > > John, > > > > You may want to ask this question on the dev list as well. > > > > I think, logically, this could be accompli

Re: help with ApacheDrill and S3

2015-09-24 Thread scott cote
comma as a delimiter, I always ALWAYS get bit in the rear. So I will switch to using a different delimiter within the log file - one that does NOT compete with the commas typically found inside JSON. You guys came through. Thank you. SCott > On Sep 23, 2015, at 9:53 AM, Andries Engelbre

Re: help with ApacheDrill and S3

2015-09-22 Thread scott cote
artifacts or if you want me to turn on a debug/verbose flag. Thanks, SCott ——>>>>>>1. First Attempt (use s3n) <<<<<<<<<<<<<< with the connections string doing the “s3n” prefix - null pointer errors Here is the co

Re: help with ApacheDrill and S3

2015-09-22 Thread scott cote
. The only xml that I saw was the core-site.xml and it does not have that kind of information within it …. Thanks in advance. SCott > On Sep 22, 2015, at 10:47 AM, Andries Engelbrecht <aengelbre...@maprtech.com> > wrote: > > Scott, > > In your SP configuration c

help with ApacheDrill and S3

2015-09-21 Thread scott cote
of my hadoop_excludes.txt 5. View of the end of the sqlline.log Please let me know what I should change? Thanks, SCott Scott C. Cote scottcc...@gmail.com <mailto:scottcc...@gmail.com> 972.672.6484 =>>>>> Part 1 <<<<= What I see when I request for a list of da

Re: Announcing new committer: Ellen Friedman

2015-11-22 Thread scott cote
Congrats Ellen. To the point, we wouldn’t have had Drill as a topic at @DfwDataScience and @DallasMongoDB if she had not evangelized with us. :) SCott > On Nov 22, 2015, at 6:55 PM, Andrew Brust > <andrew.br...@bluebadgeinsights.com> wrote: > > Excellent news! > &g

drill wont open parquet files in s3 with _metadata and _common_metadata files

2016-06-14 Thread Scott Kinney
around them in some way? Thanks all Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and ma

Re: drill wont open parquet files in s3 with _metadata and _common_metadata files

2016-06-14 Thread Scott Kinney
Nevermind, this is because I changed the directory structure. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information

Do i need hadoop installed to use dfs storage?

2016-05-31 Thread Scott Kinney
ultInputFormat": "json" }, ... And i get the same thing. I do not have hadoop installed. do i need it? Thanks, Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments c

Re: queries take over 2 min

2016-06-02 Thread Scott Kinney
Yep, I disabled all other storage plugins (default dfs, cp and custom s3 plugin) and the alter session took .5 sec as opposed to 169. Thank you! Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail

connecting to s3

2016-05-26 Thread Scott Kinney
, ResponseStatus: Forbidden, XML Error Message: AccessDeniedAccess Denied95BB171A886F3C27NLoKTo5jzzBS3j0mIRUeVwF8tGvTFUh/4jJrkPr+0U6kjo1e3t7uX3wv2Thkb1G/+/ZDODJXbVE= What am I doing wrong? Thanks all, Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.28

Re: connecting to s3

2016-05-26 Thread scott cote
My post is dated also :) Eventually got around it. but I don’t remember how at the moment. No worries SCott > On May 26, 2016, at 4:44 PM, Nathan Griffith <ngriff...@dremio.com> wrote: > > Hi, Scott. > > That article is a bit dated. Try setting up your core-site.xml fil

Re: Analyse web server logs in Drill

2016-02-06 Thread Jim Scott
il.com> wrote: > > Niels Basjes and Jim Scott worked on something related to this recently. > > The idea was to make the output of Niels' log-parser package directly > queryable in Drill. > > I don't know the current status. > > > On Sat, Feb 6, 2016 at 4:25 PM, David A

Re: Hangout Starting

2016-01-29 Thread scott cote
Anyone working on a DrillBit that can poke into an ignite grid? SCott > On Jan 26, 2016, at 11:58 AM, Jacques Nadeau <jacq...@dremio.com> wrote: > > https://plus.google.com/hangouts/_/dremio.com/drillhangout?authuser=0 > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio

Re: Hangout Starting

2016-01-29 Thread scott cote
almost always beets the pants off of pulling data from storage …. Reach out to them u...@ignite.apache.org <mailto:u...@ignite.apache.org> Might be a nice mashup for Dremio and GridGain :) Hope this helps you guys grow. SCott > On Jan 29, 2016, at 11:39 AM, Stefán Baxter <ste...@

Date Format conversion

2016-02-23 Thread Wilburn, Scott
Hello, Is there a way to convert other date formats into the DATE type? I would like to cast a field that contains a date in the format mm/dd/, i.e "01/25/2016", into DATE type. Thanks, Scott Wilburn

RE: [E] Re: Date Format conversion

2016-02-23 Thread Wilburn, Scott
; > On Tue, Feb 23, 2016 at 10:41 AM, Wilburn, Scott < > scott.wilb...@verizonwireless.com.invalid> wrote: > > > Hello, > > Is there a way to convert other date formats into the DATE type? I > > would like to cast a field that contains a date in the forma

Multiple Delimiter Format

2016-02-18 Thread Wilburn, Scott
Hello, Is there a way to specify multiple delimiters when configuring a storage plugin record format? For example, could I split records into fields by comma or by semicolon characters. Thanks, Scott Wilburn

Re: Multiple Delimiter Format

2016-02-18 Thread Jim Scott
Scott, You would need a format defined for each file type. e.g. csv has commas, tsv has tabs, so on If you are looking for multiple delimiters within the same file or potentially with a single file extension that isn't supported. Jim On Thu, Feb 18, 2016 at 5:43 PM, Wilburn, Scott < scott.w

RE: [E] Re: Multiple Delimiter Format

2016-02-18 Thread Wilburn, Scott
] = aab columns[5] = 99 columns[6] = 66 columns[7] = aab In this example, the record contains 8 fields when split by comma and by semicolon. Is something like this possible? Thanks, Scott Wilburn -Original Message- From: Jim Scott [mailto:jsc...@maprtech.com] Sent: Thursday, February 18

RE: [E] Re: Multiple Delimiter Format

2016-02-19 Thread Wilburn, Scott
Magnus, Great suggestions. I would like to try to extend the functionality of the "text" format type, used for csv files today. Do you know if there are any published instructions for this process? Thanks, Scott Wilburn -Original Message- From: Magnus Pierre [m

dfs filename

2016-03-21 Thread Wilburn, Scott
, key and value, I really want to print three columns: filename,key,value . Any help would be appreciated. Thanks, Scott Wilburn

RE: [E] Re: JDBC storage plugin fails

2016-03-24 Thread Wilburn, Scott
It is working now. Thanks for the suggestions. It was a matter of getting the user credentials working for my database. I only wish the error messages were more descriptive, and specific to the failure reason. Glad you opened that ticket, Chris. Thanks, Scott Wilburn -Original Message

Drill indexes

2016-03-25 Thread Wilburn, Scott
Hello, Is there a way to index or create primary keys for fields in Drill? I'm interested in indexing for performance and also for creating unique constraints. If there is no way to create indexes, then is there another way to generate a unique constraint? Thanks, Scott Wilburn

binary file searching

2016-03-08 Thread Wilburn, Scott
Hello, Is there a way to use Drill to search through binary files? I have some binary files that require a special compiled tool to convert into CSV format. Can Drill execute an external tool to convert each file into ASCII before searching? Thanks, Scott Wilburn

RE: [E] Re: binary file searching

2016-03-08 Thread Wilburn, Scott
The compiled tool I use to read the files today is written in C. It converts the files to ASCII CSV and prints to STDOUT. Would that work in a format plugin? Thanks, Scott Wilburn -Original Message- From: Jason Altekruse [mailto:altekruseja...@gmail.com] Sent: Tuesday, March 08, 2016

RE: Drill indexes

2016-03-30 Thread Wilburn, Scott
All, I didn't get any response on this question. Does anyone have a comment or advice for me? I didn't see anywhere in the Drill online docs that explain indexes or unique constraints. Did I miss something? Thanks, Scott Wilburn -Original Message- From: Wilburn, Scott

RE: [E] Re: dfs filename

2016-03-21 Thread Wilburn, Scott
Thanks. I've started a track on that. Do you know of any workaround in the meantime? Thanks, Scott Wilburn -Original Message- From: Abhishek Girish [mailto:abhishek.gir...@gmail.com] Sent: Monday, March 21, 2016 02:40 PM To: user Subject: [E] Re: dfs filename This is currently

Re: Node.js?

2016-04-29 Thread Jim Scott
Node already supports ODBC. Drill also has a ReST interface. Is there something more specific you had in mind? On Fri, Apr 29, 2016 at 6:47 AM, Jeff Gunther <jeff.gunt...@gmail.com> wrote: > Are there any plans for a Node.js driver? > > Jeff Gunther > -- *Jim Scott* Di

DATA_READ ERROR: Error processing input: Cannot use newline character within quoted string

2016-05-23 Thread Wilburn, Scott
-parsing the data? Thanks, Scott Wilburn

Re: Performance tuning

2016-05-17 Thread Jim Scott
Information Technology > Internet Based Communication Networks and Services (IBCN) > Ghent University - iMinds > Technologiepark Zwijnaarde 15, B-9052 Gent, Belgium > E: leandro.ordo...@intec.ugent.be, leandro.ordoneza...@ugent.be > W: www.ibcn.intec.UGent.be > > -- *Jim Scott*

Re: Discussion - "Hidden" Workspaces

2016-05-25 Thread Jim Scott
to provide a clean > interface and depiction of valuable schemas to my user via workspaces, and > this small flag, I believe would be a low impact way to do that. > > I would love discussion on this, if others would find this valuable, I will > happily make a JIRA. > > Jo

Re: [ANNOUNCE] New PMC Chair of Apache Drill

2016-05-25 Thread scott cote
Parth is a nice “bit” to add to the tool kit :) SCott > On May 25, 2016, at 11:15 AM, Abdel Hakim Deneche <adene...@maprtech.com> > wrote: > > Congrats Parth ! > > On Wed, May 25, 2016 at 9:15 AM, Zelaine Fong <zf...@maprtech.com> wrote: > >> C

deploy dockerized drill cluster

2016-07-22 Thread Scott Kinney
etting the json all_text_mode and read_numbers_as_double. I have a drill-embeded instance that seems to pick these json setting from the drill-override file just fine. Can you see what I'm doing wrong? ____ Scott Kinney | DevOps stem <http://www.stem.com/> |

Re: Error creating view

2016-07-22 Thread Scott Kinney
Yep, I didn't set the workspace 'writable': true user error as usual Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information

Re: deploy dockerized drill cluster

2016-07-25 Thread Scott Kinney
it's still not picking up the store.json* config changes The only way I can see to set these is with running ALTER SYSTEM query after drill api is up. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail

Error creating view

2016-07-20 Thread Scott Kinney
=0) dfsl is just a copy of the default dfs storage plugin. Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and propr

Best way to set schema to handle different json structures

2016-07-11 Thread Scott Kinney
will have out data in s3 in a name/year/month/day and I dont want to have to put this schema file in every directory is s3. that seems unmanageable. Is there a way to set the schema from a file before making a query via the REST api? Scott Kinney | DevOps stem <h

Re: Best way to set schema to handle different json structures

2016-07-11 Thread Scott Kinney
No i haven't. I'll give that a try. Thanks. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole

gzipped json files not named .json.gz

2016-06-28 Thread Scott Kinney
Can I have drill open gzipped json files who's names do not end in .json.gz? We have a spark job generating these files and it just dosn't want to change the name or append the .json.gz. ? Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.28

Scaling Drill with s3 plugin

2016-06-28 Thread Scott Kinney
>From what I can tell the s3 plugin must be configured via the web interface. >We really need drill to be able to scale and cluster. Has anyone found a way >to configure drill to use s3 with xml (or whatever) files rather than the web >interface? ? __

Re: gzipped json files not named .json.gz

2016-06-28 Thread Scott Kinney
Well that's a bummer but I believe it setting "defaultInputFormat": "json" doesn't seem to have any effect. ____ Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachmen

Re: Scaling Drill with s3 plugin

2016-06-28 Thread Scott Kinney
Ah, ok. Thats something. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole use of the intended

Re: deploy dockerized drill cluster

2016-07-23 Thread Scott Kinney
Hm, i must have set those another way in embeded mode. I can't see where. Those settings persist between drill restarts. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain

Re: gzipped json files not named .json.gz

2016-07-01 Thread Scott Kinney
. thanks! Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole use of the intended recipient(s). Any

array in json with mixed values (int and float)

2016-07-01 Thread Scott Kinney
There is a key 'foo': [ 99, 99.1, 99.8 ]. Is there a way get drill to handle this? Maybe treat all ints as floats? ____ Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any att

Re: array in json with mixed values (int and float)

2016-07-01 Thread Scott Kinney
! Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole use of the intended recipient(s). Any review, use or distribution that has

Re: array in json with mixed values (int and float)

2016-07-01 Thread Scott Kinney
That looks promising but didn't work. Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole use

Re: missing data in json structure when using web / api

2016-07-01 Thread Scott Kinney
Not that I know of but I'm new to drill. I've done 'alter system' for json all_text_mode & read_numbers_as_double. Do you know of a setting that might cause something like this? ____ Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Mill

missing data in json structure when using web / api

2016-07-01 Thread Scott Kinney
When i query from sqlline i can see all the data, very complicated / nested json structure but when i query with the api or the web ui a lot of the data is missing. ? Scott Kinney | DevOps stem <http://www.stem.com/> | m 510.282.1299 100 Rollin

Re: "Transactional" conversion of CSV to Parquet?

2016-10-24 Thread Jim Scott
cate > data through both an original csv and converted Parquet file at the same > time. > > Is there a common practice to making data available once converted, in > something similar to a transactional batch of "convert then (re)move source > csv files" ? > -- *Jim Scot

Re: Executing Drill queries automatically (and reactively)

2015-09-22 Thread Scott C. Cote
Sounds like an opportunity for drill triggers Sent from my iPhone > On Sep 22, 2015, at 2:18 PM, Edmon Begoli wrote: > > Yes, that is my current thinking and most likely course of action. > >> On Tue, Sep 22, 2015 at 3:13 PM, Paul Ilechko wrote: >>