Thanks for the fix Bridget. I just took a look at the posted version an noticed that the example query is missing some necessary backticks
Julien had tried to include an escaped backtick in his PR, but it doesn't look like this is the right way to include a backtick in this type of text. The path name to the file should be in backticks, like it is with a regular from clause. Current: table(dfs.path/to/data.csv(type => 'text', Correct: table(dfs.`path/to/data.csv`(type => 'text', I don't know if we frequently use these inline monospace blocks, but we might have similar issues elsewhere. As far as getting a backtick into one it seems like there are two possible solutions: http://meta.stackexchange.com/questions/55437/how-can-the-backtick-character-be-included-in-code Do you know which of these has been used previously? If we aren't using these inline blocks elsewhere, we could just replace this with the separated code/monospace blocks we normally use. On Fri, Dec 4, 2015 at 4:40 PM, Bridget Bevens <[email protected]> wrote: > Added that it's available in Drill 1.4 and later: > > http://drill.apache.org/docs/plugin-configuration-basics/#using-the-formats-attributes-as-table-function-parameters > > Thanks, > Bridget > > On Thu, Dec 3, 2015 at 9:55 PM, Jacques Nadeau <[email protected]> wrote: > > > One note, this feature is in the upcoming Drill 1.4, not 1.3 > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Thu, Dec 3, 2015 at 5:11 PM, Julien Le Dem <[email protected]> wrote: > > > > > Thank you! > > > > > > On Thu, Dec 3, 2015 at 5:08 PM, Jason Altekruse < > > [email protected]> > > > wrote: > > > > > > > Thanks Bridget! > > > > > > > > On Thu, Dec 3, 2015 at 4:14 PM, Bridget Bevens <[email protected] > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I updated the doc page on the Drill site: > > > > > > > > > > > > > > > > > > > > http://drill.apache.org/docs/plugin-configuration-basics/#using-the-formats-attributes-as-table-function-parameters > > > > > > > > > > Thanks, > > > > > Bridget > > > > > > > > > > On Thu, Dec 3, 2015 at 3:53 PM, Julien Le Dem <[email protected]> > > > wrote: > > > > > > > > > > > Here's a PR for the doc: > > > > > > https://github.com/apache/drill/pull/290 > > > > > > > > > > > > On Thu, Dec 3, 2015 at 3:25 PM, Julien Le Dem <[email protected] > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > I need to update the doc to for this. I'll send a PR soon. > > > > > > > In the meantime you can look at the tests: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/drill/blob/d855906b95d4182a93af936c4e16888a770039b5/exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java > > > > > > > Basically there is one type for each Format plugin. > > > > > > > It look at the classes that implement FormatPluginConfig just > > like > > > > for > > > > > > the > > > > > > > json based configuration: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/drill/blob/d855906b95d4182a93af936c4e16888a770039b5/logical/src/main/java/org/apache/drill/common/logical/FormatPluginConfig.java > > > > > > > > > > > > > > For example for the "text" format: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/drill/blob/d855906b95d4182a93af936c4e16888a770039b5/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/TextFormatPlugin.java#L135 > > > > > > > > > > > > > > the type is "text" as defined by the annotation:@JsonTypeName > > > > ("text") > > > > > > > the available parameters are the same fields as in the json > conf > > > with > > > > > the > > > > > > > same defaults: > > > > > > > public String lineDelimiter = "\n"; > > > > > > > public char fieldDelimiter = '\n'; > > > > > > > public char quote = '"'; > > > > > > > public char escape = '"'; > > > > > > > public char comment = '#'; > > > > > > > public boolean skipFirstLine = false; > > > > > > > public boolean extractHeader = false; > > > > > > > > > > > > > > On Thu, Dec 3, 2015 at 1:12 PM, Jason Altekruse < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> I don't think we have anything posted right now, it was just > > > merged > > > > > last > > > > > > >> week. > > > > > > >> > > > > > > >> Julien, > > > > > > >> Did you have something written for a short bit of > documentation > > on > > > > the > > > > > > >> functionality and any current limitations? > > > > > > >> > > > > > > >> - Jason > > > > > > >> > > > > > > >> On Thu, Dec 3, 2015 at 12:16 PM, Abdel Hakim Deneche < > > > > > > >> [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > I didn't notice select with options is already available !!! > > did > > > > we > > > > > > add > > > > > > >> it > > > > > > >> > to the documentation ? > > > > > > >> > > > > > > > >> > On Thu, Dec 3, 2015 at 12:05 PM, Jason Altekruse < > > > > > > >> [email protected] > > > > > > >> > > wrote: > > > > > > >> > > > > > > > >> >> Yes! > > > > > > >> >> > > > > > > >> >> Thanks to the new feature "select with options" it is > > possible > > > to > > > > > > >> >> configure > > > > > > >> >> the text reader to have query specific options. You will > need > > > to > > > > > > build > > > > > > >> the > > > > > > >> >> tip of master or use the soon to be posted release > candidate > > > for > > > > > 1.4 > > > > > > to > > > > > > >> >> use > > > > > > >> >> the feature. > > > > > > >> >> > > > > > > >> >> select a, b from table(dfs.`path/to/data.csv`(type => > 'text', > > > > > > >> >> fieldDelimiter => ',', extractHeader => true)) > > > > > > >> >> > > > > > > >> >> On Thu, Dec 3, 2015 at 11:51 AM, John Omernik < > > > [email protected]> > > > > > > >> wrote: > > > > > > >> >> > > > > > > >> >> > I can't reproduce, so I must have done something wrong > the > > > > first > > > > > > >> time, > > > > > > >> >> > thank you for replying. > > > > > > >> >> > > > > > > > >> >> > Is there away to select from a csv directory with extract > > > > header > > > > > > for > > > > > > >> >> only > > > > > > >> >> > that query or table? (Options?) > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > On Wed, Dec 2, 2015 at 11:56 AM, Abdel Hakim Deneche < > > > > > > >> >> > [email protected]> > > > > > > >> >> > wrote: > > > > > > >> >> > > > > > > > >> >> > > Hey John, > > > > > > >> >> > > > > > > > > >> >> > > What do you get when you run "select * from > sys.version" > > ? > > > > > > >> >> > > > > > > > > >> >> > > extractHeader is false by default, so you need to > > > explicitly > > > > > set > > > > > > >> it to > > > > > > >> >> > > true. > > > > > > >> >> > > > > > > > > >> >> > > can you post your storage plugin configuration ? > > > > > > >> >> > > > > > > > > >> >> > > Thanks > > > > > > >> >> > > > > > > > > >> >> > > On Tue, Dec 1, 2015 at 6:04 AM, John Omernik < > > > > [email protected] > > > > > > > > > > > > >> >> wrote: > > > > > > >> >> > > > > > > > > >> >> > > > Hey all, > > > > > > >> >> > > > > > > > > > >> >> > > > Per my comment on > > > > > > >> https://issues.apache.org/jira/browse/DRILL-4145, > > > > > > >> >> I > > > > > > >> >> > > am > > > > > > >> >> > > > curious on why a CSV query (I am assuming with a > > default > > > > > > >> >> configuration, > > > > > > >> >> > > but > > > > > > >> >> > > > I have asked the question) in S3 would interpret > > > > differently > > > > > > >> than a > > > > > > >> >> CSV > > > > > > >> >> > > > query in MaprFS. > > > > > > >> >> > > > > > > > > > >> >> > > > Per the other user, they are using Drill 1.3, and I > am > > as > > > > > well > > > > > > >> (per > > > > > > >> >> the > > > > > > >> >> > > > MapR folks, I am using a Dev release version from > MapR > > > that > > > > > has > > > > > > >> the > > > > > > >> >> > > Office > > > > > > >> >> > > > 1.3 release code base) > > > > > > >> >> > > > > > > > > > >> >> > > > Basically, The query from the JIRA author showed the > > CSV > > > > file > > > > > > >> being > > > > > > >> >> > > > interpreted, i.e. the "FIELD_1", "FIELD_2" etc were > the > > > > > headers > > > > > > >> and > > > > > > >> >> the > > > > > > >> >> > > > results broken out into columns. When I did this on > the > > > > same > > > > > > >> data, I > > > > > > >> >> > got > > > > > > >> >> > > > one results, columns and an array of data. > > > > > > >> >> > > > > > > > > > >> >> > > > I tried setting extractHeader: true (what is the > > default > > > on > > > > > > this > > > > > > >> >> > setting) > > > > > > >> >> > > > and that had no effect. (After I update a storage > > plugin, > > > > > what > > > > > > >> do I > > > > > > >> >> > need > > > > > > >> >> > > to > > > > > > >> >> > > > do to ensure I see the effect in my SQL line session? > > DO > > > I > > > > > need > > > > > > >> to > > > > > > >> >> > > > reconnect? Basically I set the storage plugin, got > the > > > > > > "success" > > > > > > >> >> then > > > > > > >> >> > > > changed to a difference schema and then back to my > > > original > > > > > > >> schema > > > > > > >> >> and > > > > > > >> >> > > saw > > > > > > >> >> > > > no effect... should I reconnect or is that not > needed?) > > > > > > >> >> > > > > > > > > > >> >> > > > Just curious on why we'd see different ways to read > CSV > > > > > files, > > > > > > >> the > > > > > > >> >> S3 > > > > > > >> >> > vs. > > > > > > >> >> > > > MapRFS shouldn't be different... or am I missing > > > something? > > > > > > >> >> > > > > > > > > > >> >> > > > Thanks! > > > > > > >> >> > > > > > > > > > >> >> > > > John > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > -- > > > > > > >> >> > > > > > > > > >> >> > > Abdelhakim Deneche > > > > > > >> >> > > > > > > > > >> >> > > Software Engineer > > > > > > >> >> > > > > > > > > >> >> > > <http://www.mapr.com/> > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > Now Available - Free Hadoop On-Demand Training > > > > > > >> >> > > < > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > -- > > > > > > >> > > > > > > > >> > Abdelhakim Deneche > > > > > > >> > > > > > > > >> > Software Engineer > > > > > > >> > > > > > > > >> > <http://www.mapr.com/> > > > > > > >> > > > > > > > >> > > > > > > > >> > Now Available - Free Hadoop On-Demand Training > > > > > > >> > < > > > > > > >> > > > > > > > > > > > > > > > > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Julien > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Julien > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Julien > > > > > >
