[jira] [Commented] (DRILL-7048) Implement JDBC Statement.setMaxRows() with System Option
[ https://issues.apache.org/jira/browse/DRILL-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835203#comment-16835203 ] Bridget Bevens commented on DRILL-7048: --- Hi [~kkhatua], I've updated the following page with info about setMaxRows: https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-drill-driver-class-name Also included it in the Note in this section: https://drill.apache.org/docs/planning-and-execution-options/#drill-web-ui-row-limit-settings Let me know if I need to make any changes. Thanks, Bridget > Implement JDBC Statement.setMaxRows() with System Option > > > Key: DRILL-7048 > URL: https://issues.apache.org/jira/browse/DRILL-7048 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC, Query Planning Optimization >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > > With DRILL-6960, the webUI will get an auto-limit on the number of results > fetched. > Since more of the plumbing is already there, it makes sense to provide the > same for the JDBC client. > In addition, it would be nice if the Server can have a pre-defined value as > well (default 0; i.e. no limit) so that an _admin_ would be able to ensure a > max limit on the resultset size as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-7048) Implement JDBC Statement.setMaxRows() with System Option
[ https://issues.apache.org/jira/browse/DRILL-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835203#comment-16835203 ] Bridget Bevens edited comment on DRILL-7048 at 5/7/19 11:59 PM: Hi [~kkhatua], I've updated the following page with info about setMaxRows: https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-drill-driver-class-name Also included it in the Note in this section: https://drill.apache.org/docs/planning-and-execution-options/#drill-web-ui-row-limit-settings Let me know if I need to make any changes. Thanks, Bridget was (Author: bbevens): Hi [~kkhatua], I've updated the following page with info about setMaxRows: https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-drill-driver-class-name Also included it in the Note in this section: https://drill.apache.org/docs/planning-and-execution-options/#drill-web-ui-row-limit-settings Let me know if I need to make any changes. Thanks, Bridget > Implement JDBC Statement.setMaxRows() with System Option > > > Key: DRILL-7048 > URL: https://issues.apache.org/jira/browse/DRILL-7048 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC, Query Planning Optimization >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > > With DRILL-6960, the webUI will get an auto-limit on the number of results > fetched. > Since more of the plumbing is already there, it makes sense to provide the > same for the JDBC client. > In addition, it would be nice if the Server can have a pre-defined value as > well (default 0; i.e. no limit) so that an _admin_ would be able to ensure a > max limit on the resultset size as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-7239) Download page wrong date
[ https://issues.apache.org/jira/browse/DRILL-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens resolved DRILL-7239. --- Resolution: Fixed Fixed Date on Download page. Will check on copyright date. > Download page wrong date > > > Key: DRILL-7239 > URL: https://issues.apache.org/jira/browse/DRILL-7239 > Project: Apache Drill > Issue Type: Bug >Reporter: Sebb >Assignee: Bridget Bevens >Priority: Major > > [https://drill.apache.org/download/] says: > > "Drill 1.16 was released on May 02, 2018." > I think that is wrong. > > The page also says: > "Copyright © 2012-2014" > If the year is mentioned, it should be updated for the last substantive change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-7235) Download page should link to verification instructions
[ https://issues.apache.org/jira/browse/DRILL-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens resolved DRILL-7235. --- Resolution: Fixed Download page updated with a link to instructions to check downloaded files. > Download page should link to verification instructions > -- > > Key: DRILL-7235 > URL: https://issues.apache.org/jira/browse/DRILL-7235 > Project: Apache Drill > Issue Type: Bug >Reporter: Sebb >Assignee: Bridget Bevens >Priority: Major > > The download page include links to KEYS, sigs and hashes, but does not > provide any information on why or how they should be used. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7235) Download page should link to verification instructions
[ https://issues.apache.org/jira/browse/DRILL-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832689#comment-16832689 ] Bridget Bevens commented on DRILL-7235: --- I've updated the [Download page|https://drill.apache.org/download/] with a link to: [How to Verify Downloaded Files|https://www.apache.org/info/verification.html] You may need to refresh the page to see the update. Thanks, Bridget > Download page should link to verification instructions > -- > > Key: DRILL-7235 > URL: https://issues.apache.org/jira/browse/DRILL-7235 > Project: Apache Drill > Issue Type: Bug >Reporter: Sebb >Assignee: Bridget Bevens >Priority: Major > > The download page include links to KEYS, sigs and hashes, but does not > provide any information on why or how they should be used. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6989) Upgrade to SqlLine 1.7.0
[
https://issues.apache.org/jira/browse/DRILL-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6989:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Upgrade to SqlLine 1.7.0
>
>
> Key: DRILL-6989
> URL: https://issues.apache.org/jira/browse/DRILL-6989
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
> Attachments: sqliine.JPG
>
>
> Upgrade to SqlLine 1.7.0 after its officially released.
> Major improvements is that Drill Sqlline prom will change from the default
> {{0: jdbc:drill:zk=local>}} to {{apache drill>}} if schema is unset. If
> schema was set (ex: {{use dfs.tmp}}), prompt will look the following way:
> {{apache drill (dfs.tmp)>}}.
> To return to previous prompt display, user can modify
> {{drill-sqlline-override.conf}} and set {{drill.sqlline.prompt.with_schema}}
> to false.
> To set custom prompt user can use one of the SqlLine properties: prompt,
> promptScript, rightPrompt.
> This can be achieved using set command, example: {{!set prompt my_prompt)}},
> or these properties can be overridden in {{drill-sqlline-override.conf}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6989) Upgrade to SqlLine 1.7.0
[
https://issues.apache.org/jira/browse/DRILL-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832062#comment-16832062
]
Bridget Bevens commented on DRILL-6989:
---
Hi [~arina],
Doc is updated with this info here:
https://drill.apache.org/docs/configuring-the-drill-shell/
Thanks,
Bridget
> Upgrade to SqlLine 1.7.0
>
>
> Key: DRILL-6989
> URL: https://issues.apache.org/jira/browse/DRILL-6989
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
> Attachments: sqliine.JPG
>
>
> Upgrade to SqlLine 1.7.0 after its officially released.
> Major improvements is that Drill Sqlline prom will change from the default
> {{0: jdbc:drill:zk=local>}} to {{apache drill>}} if schema is unset. If
> schema was set (ex: {{use dfs.tmp}}), prompt will look the following way:
> {{apache drill (dfs.tmp)>}}.
> To return to previous prompt display, user can modify
> {{drill-sqlline-override.conf}} and set {{drill.sqlline.prompt.with_schema}}
> to false.
> To set custom prompt user can use one of the SqlLine properties: prompt,
> promptScript, rightPrompt.
> This can be achieved using set command, example: {{!set prompt my_prompt)}},
> or these properties can be overridden in {{drill-sqlline-override.conf}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7058) Refresh command to support subset of columns
[ https://issues.apache.org/jira/browse/DRILL-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7058: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > Refresh command to support subset of columns > > > Key: DRILL-7058 > URL: https://issues.apache.org/jira/browse/DRILL-7058 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > Modify the REFRESH TABLE METADATA command to specify selected columns which > are deemed interesting in some form - either sorted/partitioned/clustered by > and column metadata will be stored only for those columns. The proposed > syntax is > REFRESH TABLE METADATA *_[ COLUMNS (list of columns) / NONE ]_* path> > For example, REFRESH TABLE METADATA COLUMNS (age, salary) `/tmp/employee` > stores column metadata only for the age and salary columns. REFRESH TABLE > METADATA COLUMNS NONE `/tmp/employee` will not store column metadata for any > column. > By default, if the optional 'COLUMNS' clause is omitted, column metadata is > collected for all the columns. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7058) Refresh command to support subset of columns
[ https://issues.apache.org/jira/browse/DRILL-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832061#comment-16832061 ] Bridget Bevens commented on DRILL-7058: --- Hi [~vdonapati], Doc is posted here: https://drill.apache.org/docs/refresh-table-metadata/ Thanks, Bridget > Refresh command to support subset of columns > > > Key: DRILL-7058 > URL: https://issues.apache.org/jira/browse/DRILL-7058 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > Modify the REFRESH TABLE METADATA command to specify selected columns which > are deemed interesting in some form - either sorted/partitioned/clustered by > and column metadata will be stored only for those columns. The proposed > syntax is > REFRESH TABLE METADATA *_[ COLUMNS (list of columns) / NONE ]_* path> > For example, REFRESH TABLE METADATA COLUMNS (age, salary) `/tmp/employee` > stores column metadata only for the age and salary columns. REFRESH TABLE > METADATA COLUMNS NONE `/tmp/employee` will not store column metadata for any > column. > By default, if the optional 'COLUMNS' clause is omitted, column metadata is > collected for all the columns. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-1328) Support table statistics
[ https://issues.apache.org/jira/browse/DRILL-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-1328: -- Labels: doc-complete (was: doc-impacting) > Support table statistics > > > Key: DRILL-1328 > URL: https://issues.apache.org/jira/browse/DRILL-1328 > Project: Apache Drill > Issue Type: Improvement >Reporter: Cliff Buchanan >Assignee: Gautam Parai >Priority: Major > Labels: doc-complete > Fix For: 1.16.0 > > Attachments: 0001-PRE-Set-value-count-in-splitAndTransfer.patch > > > This consists of several subtasks > * implement operators to generate statistics > * add "analyze table" support to parser/planner > * create a metadata provider to allow statistics to be used by optiq in > planning optimization > * implement statistics functions > Right now, the bulk of this functionality is implemented, but it hasn't been > rigorously tested and needs to have some definite answers for some of the > parts "around the edges" (how analyze table figures out where the table > statistics are located, how a table "append" should work in a read only file > system) > Also, here are a few known caveats: > * table statistics are collected by creating a sql query based on the string > path of the table. This should probably be done with a Table reference. > * Case sensitivity for column statistics is probably iffy > * Math for combining two column NDVs into a joint NDV should be checked. > * Schema changes aren't really being considered yet. > * adding getDrillTable is probably unnecessary; it might be better to do > getTable().unwrap(DrillTable.class) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-1328) Support table statistics
[ https://issues.apache.org/jira/browse/DRILL-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832059#comment-16832059 ] Bridget Bevens commented on DRILL-1328: --- Hi [~gparai], The doc is posted here: https://drill.apache.org/docs/analyze-table/ Thanks, Bridget > Support table statistics > > > Key: DRILL-1328 > URL: https://issues.apache.org/jira/browse/DRILL-1328 > Project: Apache Drill > Issue Type: Improvement >Reporter: Cliff Buchanan >Assignee: Gautam Parai >Priority: Major > Labels: doc-impacting > Fix For: 1.16.0 > > Attachments: 0001-PRE-Set-value-count-in-splitAndTransfer.patch > > > This consists of several subtasks > * implement operators to generate statistics > * add "analyze table" support to parser/planner > * create a metadata provider to allow statistics to be used by optiq in > planning optimization > * implement statistics functions > Right now, the bulk of this functionality is implemented, but it hasn't been > rigorously tested and needs to have some definite answers for some of the > parts "around the edges" (how analyze table figures out where the table > statistics are located, how a table "append" should work in a read only file > system) > Also, here are a few known caveats: > * table statistics are collected by creating a sql query based on the string > path of the table. This should probably be done with a Table reference. > * Case sensitivity for column statistics is probably iffy > * Math for combining two column NDVs into a joint NDV should be checked. > * Schema changes aren't really being considered yet. > * adding getDrillTable is probably unnecessary; it might be better to do > getTable().unwrap(DrillTable.class) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6050) Provide a limit to number of rows fetched for a query in UI
[ https://issues.apache.org/jira/browse/DRILL-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831300#comment-16831300 ] Bridget Bevens commented on DRILL-6050: --- Hi [~kkhatua], I added autoLimit to the Request Body example. Thanks, Bridget > Provide a limit to number of rows fetched for a query in UI > --- > > Key: DRILL-6050 > URL: https://issues.apache.org/jira/browse/DRILL-6050 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Labels: doc-complete, ready-to-commit, user-experience > Fix For: 1.16.0, 1.17.0 > > > Currently, the WebServer side needs to process the entire set of results and > stream it back to the WebClient. > Since the WebUI does paginate results, we can load a larger set for > pagination on the browser client and relieve pressure off the WebServer to > host all the data. > e.g. Fetching all rows from a 1Billion records table is impractical and can > be capped at 10K. Currently, the user has to explicitly specify LIMIT in the > submitted query. > An option can be provided in the field to allow for this entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6879) Indicate a warning in the WebUI when a query makes little to no progress for a while
[
https://issues.apache.org/jira/browse/DRILL-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831291#comment-16831291
]
Bridget Bevens commented on DRILL-6879:
---
Hi [~kkhatua],
I added the spacing in the icon column. It looks better - I think.
Thanks,
Bridget
> Indicate a warning in the WebUI when a query makes little to no progress for
> a while
>
>
> Key: DRILL-6879
> URL: https://issues.apache.org/jira/browse/DRILL-6879
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring, Web Server
>Affects Versions: 1.14.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
> Attachments: image-2018-12-04-11-54-54-247.png,
> image-2018-12-06-11-19-00-339.png, image-2018-12-06-11-27-14-719.png
>
>
> When running a very large query on a cluster with limited resource, we
> noticed that one of the node's VM thread freezes the fragment threads as it
> tries to do some work (GC perhaps?). This is a clear indication that the
> query is stuck in a weird state where it might not recover from.
> Under such circumstances, it makes sense to cancel or atleast warn the user
> on that page of the query exceeding a certain threshold.
> For detecting this, the user will find that the {{Last Progress}} column in
> the Fragments Overview section will show large times.
> !image-2018-12-04-11-54-54-247.png|width=969,height=336!
> In addition, there are instances where a query might have buffered operators
> spilling to disk, which also hits performance (and, subsequently, longer run
> times). Calling out this skew can be very useful.
> !image-2018-12-06-11-27-14-719.png|width=969,height=256!
> Or there might be cases where a single fragment takes much longer than the
> average (indicated by an extreme skew in the Gantt chart).
> !image-2018-12-06-11-19-00-339.png|width=969,height=150!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6964) Implement CREATE / DROP TABLE SCHEMA commands
[ https://issues.apache.org/jira/browse/DRILL-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6964: -- Labels: doc-complete ready-to-commit (was: ready-to-commit) > Implement CREATE / DROP TABLE SCHEMA commands > - > > Key: DRILL-6964 > URL: https://issues.apache.org/jira/browse/DRILL-6964 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > > Design doc - > https://docs.google.com/document/d/1mp4egSbNs8jFYRbPVbm_l0Y5GjH3HnoqCmOpMTR_g4w/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6964) Implement CREATE / DROP TABLE SCHEMA commands
[ https://issues.apache.org/jira/browse/DRILL-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829698#comment-16829698 ] Bridget Bevens commented on DRILL-6964: --- doc posted here: https://drill.apache.org/docs/create-or-replace-schema/ > Implement CREATE / DROP TABLE SCHEMA commands > - > > Key: DRILL-6964 > URL: https://issues.apache.org/jira/browse/DRILL-6964 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > > Design doc - > https://docs.google.com/document/d/1mp4egSbNs8jFYRbPVbm_l0Y5GjH3HnoqCmOpMTR_g4w/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6050) Provide a limit to number of rows fetched for a query in UI
[ https://issues.apache.org/jira/browse/DRILL-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821461#comment-16821461 ] Bridget Bevens commented on DRILL-6050: --- Hi [~kkhatua], I added the info about autoLimit to the following pages: https://drill.apache.org/docs/planning-and-execution-options/#setting-an-auto-limit-on-the-number-of-rows-returned-for-result-sets Included a note in the following section about REST client config: https://drill.apache.org/docs/planning-and-execution-options/#drill-web-ui-row-limit-settings Added the autoLimit parameter and example here: https://drill.apache.org/docs/rest-api-introduction/#post-query-json Added a note to the Limit clause: https://drill.apache.org/docs/limit-clause/ Please let me know if I need to change anything. Thanks, Bridget > Provide a limit to number of rows fetched for a query in UI > --- > > Key: DRILL-6050 > URL: https://issues.apache.org/jira/browse/DRILL-6050 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Labels: doc-impacting, ready-to-commit, user-experience > Fix For: 1.16.0, 1.17.0 > > > Currently, the WebServer side needs to process the entire set of results and > stream it back to the WebClient. > Since the WebUI does paginate results, we can load a larger set for > pagination on the browser client and relieve pressure off the WebServer to > host all the data. > e.g. Fetching all rows from a 1Billion records table is impractical and can > be capped at 10K. Currently, the user has to explicitly specify LIMIT in the > submitted query. > An option can be provided in the field to allow for this entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6050) Provide a limit to number of rows fetched for a query in UI
[ https://issues.apache.org/jira/browse/DRILL-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6050: -- Labels: doc-complete ready-to-commit user-experience (was: doc-impacting ready-to-commit user-experience) > Provide a limit to number of rows fetched for a query in UI > --- > > Key: DRILL-6050 > URL: https://issues.apache.org/jira/browse/DRILL-6050 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Labels: doc-complete, ready-to-commit, user-experience > Fix For: 1.16.0, 1.17.0 > > > Currently, the WebServer side needs to process the entire set of results and > stream it back to the WebClient. > Since the WebUI does paginate results, we can load a larger set for > pagination on the browser client and relieve pressure off the WebServer to > host all the data. > e.g. Fetching all rows from a 1Billion records table is impractical and can > be capped at 10K. Currently, the user has to explicitly specify LIMIT in the > submitted query. > An option can be provided in the field to allow for this entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6562) Plugin Management improvements
[ https://issues.apache.org/jira/browse/DRILL-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6562: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > Plugin Management improvements > -- > > Key: DRILL-6562 > URL: https://issues.apache.org/jira/browse/DRILL-6562 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Web Server >Affects Versions: 1.14.0 >Reporter: Abhishek Girish >Assignee: Vitalii Diravka >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > Attachments: Export.png, ExportAll.png, Screenshot from 2019-03-21 > 01-18-17.png, Screenshot from 2019-03-21 02-52-50.png, Storage.png, > UpdateExport.png, create.png, image-2018-07-23-02-55-02-024.png, > image-2018-10-22-20-20-24-658.png, image-2018-10-22-20-20-59-105.png > > > Follow-up to DRILL-4580. > Provide ability to export all storage plugin configurations at once, with a > new "Export All" option on the Storage page of the Drill web UI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6562) Plugin Management improvements
[ https://issues.apache.org/jira/browse/DRILL-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821460#comment-16821460 ] Bridget Bevens commented on DRILL-6562: --- Hi [~vitalii], I added info about exporting all storage plugins here: https://drill.apache.org/docs/configuring-storage-plugins/#exporting-storage-plugin-configurations I’m still not sure how the .conf file works – do you copy and paste the configurations into each storage plugin configuration manually, or is there an easier way to import the file? Thanks, Bridget > Plugin Management improvements > -- > > Key: DRILL-6562 > URL: https://issues.apache.org/jira/browse/DRILL-6562 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Web Server >Affects Versions: 1.14.0 >Reporter: Abhishek Girish >Assignee: Vitalii Diravka >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > Attachments: Export.png, ExportAll.png, Screenshot from 2019-03-21 > 01-18-17.png, Screenshot from 2019-03-21 02-52-50.png, Storage.png, > UpdateExport.png, create.png, image-2018-07-23-02-55-02-024.png, > image-2018-10-22-20-20-24-658.png, image-2018-10-22-20-20-59-105.png > > > Follow-up to DRILL-4580. > Provide ability to export all storage plugin configurations at once, with a > new "Export All" option on the Storage page of the Drill web UI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6921) Add button to reset Options filter
[ https://issues.apache.org/jira/browse/DRILL-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6921: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > Add button to reset Options filter > -- > > Key: DRILL-6921 > URL: https://issues.apache.org/jira/browse/DRILL-6921 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Arina Ielchiieva >Assignee: Kunal Khatua >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > > Currently we have ability to search options or use quick filter on Web UI. To > reset the filter, user needs to delete input from the search pane manually. > It would be nice if we had Reset button. > Also we can consider leaving filter after option update / reset rather then > reloading page without filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6921) Add button to reset Options filter
[ https://issues.apache.org/jira/browse/DRILL-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821457#comment-16821457 ] Bridget Bevens commented on DRILL-6921: --- Hi [~kkhatua], Add content about resetting filter options here, in the Search field section: https://drill.apache.org/docs/planning-and-execution-options/#setting-options-from-the-drill-web-ui Let me know if I need to make any changes. Thanks, Bridget > Add button to reset Options filter > -- > > Key: DRILL-6921 > URL: https://issues.apache.org/jira/browse/DRILL-6921 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Arina Ielchiieva >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > > Currently we have ability to search options or use quick filter on Web UI. To > reset the filter, user needs to delete input from the search pane manually. > It would be nice if we had Reset button. > Also we can consider leaving filter after option update / reset rather then > reloading page without filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6939) Indicate when a query is submitted and is in progress
[ https://issues.apache.org/jira/browse/DRILL-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821454#comment-16821454 ] Bridget Bevens commented on DRILL-6939: --- Hi [~kkhatua], I added content about the pop-up here (may need to resize image to make smaller): https://drill.apache.org/docs/starting-the-web-ui/#running-queries-from-the-web-ui Let me know if I need to change anything. Thanks, Bridget > Indicate when a query is submitted and is in progress > - > > Key: DRILL-6939 > URL: https://issues.apache.org/jira/browse/DRILL-6939 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.14.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Critical > Labels: doc-impacting, ready-to-commit, user-experience > Fix For: 1.16.0 > > > When submitting a long running query, the web UI shows no indication of the > query having been submitted. What is needed is some form of UI enhancement > that shows that the submitted query is in progress and the results will load > when available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6939) Indicate when a query is submitted and is in progress
[ https://issues.apache.org/jira/browse/DRILL-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6939: -- Labels: doc-complete ready-to-commit user-experience (was: doc-impacting ready-to-commit user-experience) > Indicate when a query is submitted and is in progress > - > > Key: DRILL-6939 > URL: https://issues.apache.org/jira/browse/DRILL-6939 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.14.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Critical > Labels: doc-complete, ready-to-commit, user-experience > Fix For: 1.16.0 > > > When submitting a long running query, the web UI shows no indication of the > query having been submitted. What is needed is some form of UI enhancement > that shows that the submitted query is in progress and the results will load > when available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
[ https://issues.apache.org/jira/browse/DRILL-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821452#comment-16821452 ] Bridget Bevens commented on DRILL-6942: --- hi [~kkhatua]. I added content about sorting columns here: https://drill.apache.org/docs/query-profiles/#viewing-a-query-profile Let me know if I need to make any changes. Thanks, Bridget > Provide ability to sort list of profiles on Drill web UI > > > Key: DRILL-6942 > URL: https://issues.apache.org/jira/browse/DRILL-6942 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Khurram Faraaz >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit, user-experience > Fix For: 1.16.0 > > > We need to provide a way to sort the query profiles, on the profiles tab on > Drill web UI. > The use case is when users run many queries (several hundred queries), they > want a way to list the queries that have taken the longest time (i.e. > duration) to complete. All queries that completed, failed or were canceled > and took very long, all such queries should be listed on the top in the > profiles tab page on Drill web ui. > An option to list the query profiles based on total duration should be > provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
[ https://issues.apache.org/jira/browse/DRILL-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6942: -- Labels: doc-complete ready-to-commit user-experience (was: doc-impacting ready-to-commit user-experience) > Provide ability to sort list of profiles on Drill web UI > > > Key: DRILL-6942 > URL: https://issues.apache.org/jira/browse/DRILL-6942 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Khurram Faraaz >Assignee: Kunal Khatua >Priority: Major > Labels: doc-complete, ready-to-commit, user-experience > Fix For: 1.16.0 > > > We need to provide a way to sort the query profiles, on the profiles tab on > Drill web UI. > The use case is when users run many queries (several hundred queries), they > want a way to list the queries that have taken the longest time (i.e. > duration) to complete. All queries that completed, failed or were canceled > and took very long, all such queries should be listed on the top in the > profiles tab page on Drill web ui. > An option to list the query profiles based on total duration should be > provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6879) Indicate a warning in the WebUI when a query makes little to no progress for a while
[
https://issues.apache.org/jira/browse/DRILL-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6879:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Indicate a warning in the WebUI when a query makes little to no progress for
> a while
>
>
> Key: DRILL-6879
> URL: https://issues.apache.org/jira/browse/DRILL-6879
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring, Web Server
>Affects Versions: 1.14.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
> Attachments: image-2018-12-04-11-54-54-247.png,
> image-2018-12-06-11-19-00-339.png, image-2018-12-06-11-27-14-719.png
>
>
> When running a very large query on a cluster with limited resource, we
> noticed that one of the node's VM thread freezes the fragment threads as it
> tries to do some work (GC perhaps?). This is a clear indication that the
> query is stuck in a weird state where it might not recover from.
> Under such circumstances, it makes sense to cancel or atleast warn the user
> on that page of the query exceeding a certain threshold.
> For detecting this, the user will find that the {{Last Progress}} column in
> the Fragments Overview section will show large times.
> !image-2018-12-04-11-54-54-247.png|width=969,height=336!
> In addition, there are instances where a query might have buffered operators
> spilling to disk, which also hits performance (and, subsequently, longer run
> times). Calling out this skew can be very useful.
> !image-2018-12-06-11-27-14-719.png|width=969,height=256!
> Or there might be cases where a single fragment takes much longer than the
> average (indicated by an extreme skew in the Gantt chart).
> !image-2018-12-06-11-19-00-339.png|width=969,height=150!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6879) Indicate a warning in the WebUI when a query makes little to no progress for a while
[
https://issues.apache.org/jira/browse/DRILL-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821451#comment-16821451
]
Bridget Bevens commented on DRILL-6879:
---
Hi [~kkhatua],
I added content about the warnings is here:
https://drill.apache.org/docs/query-profiles/#query-profile-warnings
You've reviewed this content already, but please let me know if you want me to
make any changes.
Thanks,
Bridget
> Indicate a warning in the WebUI when a query makes little to no progress for
> a while
>
>
> Key: DRILL-6879
> URL: https://issues.apache.org/jira/browse/DRILL-6879
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring, Web Server
>Affects Versions: 1.14.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
> Attachments: image-2018-12-04-11-54-54-247.png,
> image-2018-12-06-11-19-00-339.png, image-2018-12-06-11-27-14-719.png
>
>
> When running a very large query on a cluster with limited resource, we
> noticed that one of the node's VM thread freezes the fragment threads as it
> tries to do some work (GC perhaps?). This is a clear indication that the
> query is stuck in a weird state where it might not recover from.
> Under such circumstances, it makes sense to cancel or atleast warn the user
> on that page of the query exceeding a certain threshold.
> For detecting this, the user will find that the {{Last Progress}} column in
> the Fragments Overview section will show large times.
> !image-2018-12-04-11-54-54-247.png|width=969,height=336!
> In addition, there are instances where a query might have buffered operators
> spilling to disk, which also hits performance (and, subsequently, longer run
> times). Calling out this skew can be very useful.
> !image-2018-12-06-11-27-14-719.png|width=969,height=256!
> Or there might be cases where a single fragment takes much longer than the
> average (indicated by an extreme skew in the Gantt chart).
> !image-2018-12-06-11-19-00-339.png|width=969,height=150!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6562) Plugin Management improvements
[ https://issues.apache.org/jira/browse/DRILL-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819625#comment-16819625 ] Bridget Bevens commented on DRILL-6562: --- Hi [~vitalii], If you use the Export All button to export all storage plugin configurations to HOCON format (.conf file), how do you import those configurations back into Drill? My guess is that you would name the file storage-plugins-override.conf and then copy it to the /conf directory, but I'm not sure. Thanks, Bridget > Plugin Management improvements > -- > > Key: DRILL-6562 > URL: https://issues.apache.org/jira/browse/DRILL-6562 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Web Server >Affects Versions: 1.14.0 >Reporter: Abhishek Girish >Assignee: Vitalii Diravka >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > Attachments: Export.png, ExportAll.png, Screenshot from 2019-03-21 > 01-18-17.png, Screenshot from 2019-03-21 02-52-50.png, Storage.png, > UpdateExport.png, create.png, image-2018-07-23-02-55-02-024.png, > image-2018-10-22-20-20-24-658.png, image-2018-10-22-20-20-59-105.png > > > Follow-up to DRILL-4580. > Provide ability to export all storage plugin configurations at once, with a > new "Export All" option on the Storage page of the Drill web UI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-540) Allow querying hive views in Drill
[
https://issues.apache.org/jira/browse/DRILL-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819478#comment-16819478
]
Bridget Bevens commented on DRILL-540:
--
Hi [~IhorHuzenko]
I've updated this page with the info:
https://drill.apache.org/docs/querying-hive/
I've also updated this page to state that Hive views are supported as of Drill
1.16:
https://drill.apache.org/docs/querying-the-information-schema/
Please let me know if I need to change anything.
Thanks,
Bridget
> Allow querying hive views in Drill
> --
>
> Key: DRILL-540
> URL: https://issues.apache.org/jira/browse/DRILL-540
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Hive
>Reporter: Ramana Inukonda Nagaraj
>Assignee: Igor Guzenko
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently hive views cannot be queried from drill.
> This Jira aims to add support for Hive views in Drill.
> *Implementation details:*
> # Drill persists it's views metadata in file with suffix .view.drill using
> json format. For example:
> {noformat}
> {
> "name" : "view_from_calcite_1_4",
> "sql" : "SELECT * FROM `cp`.`store.json`WHERE `store_id` = 0",
> "fields" : [ {
> "name" : "*",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ "dfs", "tmp" ]
> }
> {noformat}
> Later Drill parses the metadata and uses it to treat view names in SQL as a
> subquery.
> 2. In Apache Hive metadata about views is stored in similar way to
> tables. Below is example from metastore.TBLS :
>
> {noformat}
> TBL_ID |CREATE_TIME |DB_ID |LAST_ACCESS_TIME |OWNER |RETENTION |SD_ID
> |TBL_NAME |TBL_TYPE |VIEW_EXPANDED_TEXT |
> ---||--|-|--|--|--|--|--|---|
> 2 |1542111078 |1 |0|mapr |0 |2 |cview
>|VIRTUAL_VIEW |SELECT COUNT(*) FROM `default`.`customers` |
> {noformat}
> 3. So in Hive metastore views are considered as tables of special type.
> And main benefit is that we also have expanded SQL definition of views (just
> like in view.drill files). Also reading of the metadata is already
> implemented in Drill with help of thrift Metastore API.
> 4. To enable querying of Hive views we'll reuse existing code for Drill
> views as much as possible. First in *_HiveSchemaFactory.getDrillTable_* for
> _*HiveReadEntry*_ we'll convert the metadata to instance of _*View*_ (_which
> is actually model for data persisted in .view.drill files_) and then based on
> this instance return new _*DrillViewTable*_. Using this approach drill will
> handle hive views the same way as if it was initially defined in Drill and
> persisted in .view.drill file.
> 5. For conversion of Hive types: from _*FieldSchema*_ to _*RelDataType*_
> we'll reuse existing code from _*DrillHiveTable*_, so the conversion
> functionality will be extracted and used for both (table and view) fields
> type conversions.
> *Security implications*
> Consider simple example case where we have users,
> {code:java}
> user0 user1 user2
>\ /
> group12
> {code}
> and sample db where object names contains user or group who should access
> them
> {code:java}
> db_all
> tbl_user0
> vw_user0
> tbl_group12
> vw_group12
> {code}
> There are two Hive authorization modes supported by Drill - SQL Standart and
> Strorage Based authorization. For SQL Standart authorization permissions
> were granted using SQL:
> {code:java}
> SET ROLE admin;
> GRANT SELECT ON db_all.tbl_user0 TO USER user0;
> GRANT SELECT ON db_all.vw_user0 TO USER user0;
> CREATE ROLE group12;
> GRANT ROLE group12 TO USER user1;
> GRANT ROLE group12 TO USER user2;
> GRANT SELECT ON db_all.tbl_group12 TO ROLE group12;
> GRANT SELECT ON db_all.vw_group12 TO ROLE group12;
> {code}
> And for Storage based authorization permissions were granted using commands:
> {code:java}
> hadoop fs -chown user0:user0 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 700 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 750 /user/hive/warehouse/db_all.db/tbl_group12
> hadoop fs -chown user1:group12
> /user/hive/warehouse/db_all.db/tbl_group12{code}
> Then the following table shows us results of queries for both authorization
> models.
>
> *SQL Standart | Storage Based
> Authorization*
> ||SQL||user0||user1||user2|| ||user0||user1||user2||
> |*Queries executed using Drill :*| | | | | | | |
> |SHOW TABLES IN hive.db_all;| all| all| all| |Accessibe tables +
[jira] [Updated] (DRILL-540) Allow querying hive views in Drill
[
https://issues.apache.org/jira/browse/DRILL-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-540:
-
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Allow querying hive views in Drill
> --
>
> Key: DRILL-540
> URL: https://issues.apache.org/jira/browse/DRILL-540
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Hive
>Reporter: Ramana Inukonda Nagaraj
>Assignee: Igor Guzenko
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently hive views cannot be queried from drill.
> This Jira aims to add support for Hive views in Drill.
> *Implementation details:*
> # Drill persists it's views metadata in file with suffix .view.drill using
> json format. For example:
> {noformat}
> {
> "name" : "view_from_calcite_1_4",
> "sql" : "SELECT * FROM `cp`.`store.json`WHERE `store_id` = 0",
> "fields" : [ {
> "name" : "*",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ "dfs", "tmp" ]
> }
> {noformat}
> Later Drill parses the metadata and uses it to treat view names in SQL as a
> subquery.
> 2. In Apache Hive metadata about views is stored in similar way to
> tables. Below is example from metastore.TBLS :
>
> {noformat}
> TBL_ID |CREATE_TIME |DB_ID |LAST_ACCESS_TIME |OWNER |RETENTION |SD_ID
> |TBL_NAME |TBL_TYPE |VIEW_EXPANDED_TEXT |
> ---||--|-|--|--|--|--|--|---|
> 2 |1542111078 |1 |0|mapr |0 |2 |cview
>|VIRTUAL_VIEW |SELECT COUNT(*) FROM `default`.`customers` |
> {noformat}
> 3. So in Hive metastore views are considered as tables of special type.
> And main benefit is that we also have expanded SQL definition of views (just
> like in view.drill files). Also reading of the metadata is already
> implemented in Drill with help of thrift Metastore API.
> 4. To enable querying of Hive views we'll reuse existing code for Drill
> views as much as possible. First in *_HiveSchemaFactory.getDrillTable_* for
> _*HiveReadEntry*_ we'll convert the metadata to instance of _*View*_ (_which
> is actually model for data persisted in .view.drill files_) and then based on
> this instance return new _*DrillViewTable*_. Using this approach drill will
> handle hive views the same way as if it was initially defined in Drill and
> persisted in .view.drill file.
> 5. For conversion of Hive types: from _*FieldSchema*_ to _*RelDataType*_
> we'll reuse existing code from _*DrillHiveTable*_, so the conversion
> functionality will be extracted and used for both (table and view) fields
> type conversions.
> *Security implications*
> Consider simple example case where we have users,
> {code:java}
> user0 user1 user2
>\ /
> group12
> {code}
> and sample db where object names contains user or group who should access
> them
> {code:java}
> db_all
> tbl_user0
> vw_user0
> tbl_group12
> vw_group12
> {code}
> There are two Hive authorization modes supported by Drill - SQL Standart and
> Strorage Based authorization. For SQL Standart authorization permissions
> were granted using SQL:
> {code:java}
> SET ROLE admin;
> GRANT SELECT ON db_all.tbl_user0 TO USER user0;
> GRANT SELECT ON db_all.vw_user0 TO USER user0;
> CREATE ROLE group12;
> GRANT ROLE group12 TO USER user1;
> GRANT ROLE group12 TO USER user2;
> GRANT SELECT ON db_all.tbl_group12 TO ROLE group12;
> GRANT SELECT ON db_all.vw_group12 TO ROLE group12;
> {code}
> And for Storage based authorization permissions were granted using commands:
> {code:java}
> hadoop fs -chown user0:user0 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 700 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 750 /user/hive/warehouse/db_all.db/tbl_group12
> hadoop fs -chown user1:group12
> /user/hive/warehouse/db_all.db/tbl_group12{code}
> Then the following table shows us results of queries for both authorization
> models.
>
> *SQL Standart | Storage Based
> Authorization*
> ||SQL||user0||user1||user2|| ||user0||user1||user2||
> |*Queries executed using Drill :*| | | | | | | |
> |SHOW TABLES IN hive.db_all;| all| all| all| |Accessibe tables + all
> views|Accessibe tables + all views|Accessibe tables + all views|
> |SELECT * FROM hive.db_all.tbl_user0;| (/)| (x)| (x)| | (/)|
> (x)| (x)|
> |SELECT * FROM hive.db_all.vw_user0;| (/)| (x)| (x)| | (/)|
> (x)| (x)|
>
[jira] [Updated] (DRILL-7177) Format Plugin for Excel Files
[ https://issues.apache.org/jira/browse/DRILL-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7177: -- Labels: doc-impacting (was: ) > Format Plugin for Excel Files > - > > Key: DRILL-7177 > URL: https://issues.apache.org/jira/browse/DRILL-7177 > Project: Apache Drill > Issue Type: Improvement >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Labels: doc-impacting > Fix For: 1.17.0 > > > This pull request adds the functionality which enables Drill to query > Microsoft Excel files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7110) Skip writing profile when an ALTER SESSION is executed
[
https://issues.apache.org/jira/browse/DRILL-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818566#comment-16818566
]
Bridget Bevens commented on DRILL-7110:
---
Hi [~kkhatua],
I've updated the following page to include the information about this change:
https://drill.apache.org/docs/set/#usage-notes
Setting doc to complete. Please let me know if I need to make any changes.
Thanks,
Bridget
> Skip writing profile when an ALTER SESSION is executed
> --
>
> Key: DRILL-7110
> URL: https://issues.apache.org/jira/browse/DRILL-7110
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently, any {{ALTER }} query will be logged. While this is useful,
> it can potentially add up to a lot of profiles being written unnecessarily,
> since those changes are also reflected on the queries that follow.
> This JIRA is proposing an option to skip writing such profiles to the profile
> store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7110) Skip writing profile when an ALTER SESSION is executed
[
https://issues.apache.org/jira/browse/DRILL-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7110:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Skip writing profile when an ALTER SESSION is executed
> --
>
> Key: DRILL-7110
> URL: https://issues.apache.org/jira/browse/DRILL-7110
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently, any {{ALTER }} query will be logged. While this is useful,
> it can potentially add up to a lot of profiles being written unnecessarily,
> since those changes are also reflected on the queries that follow.
> This JIRA is proposing an option to skip writing such profiles to the profile
> store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7038:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL: https://issues.apache.org/jira/browse/DRILL-7038
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> For tables with hive-style partitions like
> {code}
> /table/2018/Q1
> /table/2018/Q2
> /table/2019/Q1
> etc.
> {code}
> if any of the following queries is run:
> {code}
> select distinct dir0 from dfs.`/table`
> {code}
> {code}
> select dir0 from dfs.`/table` group by dir0
> {code}
> it will actually scan every single record in the table rather than just
> getting a list of directories at the dir0 level. This applies even when
> cached metadata is available. This is a big penalty especially as the
> datasets grow.
> To avoid such situations, a logical prune rule can be used to collect
> partition columns (`dir0`), either from metadata cache (if available) or
> group scan, and drop unnecessary files from being read. The rule will be
> applied on following conditions:
> 1) all queried columns are partitoin columns, and
> 2) either {{DISTINCT}} or {{GROUP BY}} operations are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818564#comment-16818564
]
Bridget Bevens commented on DRILL-7038:
---
Hi [~KazydubB],
I changed the text based on your feedback. I've updated the following page with
the information:
https://drill.apache.org/docs/querying-directories/#querying-partitioned-directories
Setting doc status to complete, but please let me know if I need to make any
changes.
Thanks,
Bridget
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL: https://issues.apache.org/jira/browse/DRILL-7038
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> For tables with hive-style partitions like
> {code}
> /table/2018/Q1
> /table/2018/Q2
> /table/2019/Q1
> etc.
> {code}
> if any of the following queries is run:
> {code}
> select distinct dir0 from dfs.`/table`
> {code}
> {code}
> select dir0 from dfs.`/table` group by dir0
> {code}
> it will actually scan every single record in the table rather than just
> getting a list of directories at the dir0 level. This applies even when
> cached metadata is available. This is a big penalty especially as the
> datasets grow.
> To avoid such situations, a logical prune rule can be used to collect
> partition columns (`dir0`), either from metadata cache (if available) or
> group scan, and drop unnecessary files from being read. The rule will be
> applied on following conditions:
> 1) all queried columns are partitoin columns, and
> 2) either {{DISTINCT}} or {{GROUP BY}} operations are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7063) Create separate summary file for schema, totalRowCount, totalNullCount (includes maintenance)
[ https://issues.apache.org/jira/browse/DRILL-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816730#comment-16816730 ] Bridget Bevens commented on DRILL-7063: --- Hi [~vvysotskyi], The doc is currently here: https://docs.google.com/document/d/1V9VT8r1MRnyoQfraVY72uZPGkLuS7Gm62KM5eTuAwyA/edit?usp=sharing Thanks, Bridget > Create separate summary file for schema, totalRowCount, totalNullCount > (includes maintenance) > - > > Key: DRILL-7063 > URL: https://issues.apache.org/jira/browse/DRILL-7063 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > Original Estimate: 252h > Remaining Estimate: 252h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens resolved DRILL-7071. --- Resolution: Fixed Hi [~vitalii], I've updated the following page based on the reserved keyword in the PRs listed: https://drill.apache.org/docs/reserved-keywords/ Please let me know if you see any issues. Thanks, Bridget > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Bridget Bevens >Priority: Minor > Labels: doc, doc-impacting, documentation, keyword, reserved-word > Fix For: Future > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7071: -- Labels: doc doc-complete documentation keyword reserved-word (was: doc doc-impacting documentation keyword reserved-word) > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Bridget Bevens >Priority: Minor > Labels: doc, doc-complete, documentation, keyword, reserved-word > Fix For: Future > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7071: -- Labels: doc doc-impacting documentation keyword reserved-word (was: doc documentation keyword reserved-word) > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Bridget Bevens >Priority: Minor > Labels: doc, doc-impacting, documentation, keyword, reserved-word > Fix For: Future > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-5658) Documentation for Drill Crypto Functions
[ https://issues.apache.org/jira/browse/DRILL-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens closed DRILL-5658. - doc posted > Documentation for Drill Crypto Functions > > > Key: DRILL-5658 > URL: https://issues.apache.org/jira/browse/DRILL-5658 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Charles Givre >Assignee: Bridget Bevens >Priority: Major > Labels: doc-complete > Fix For: 1.16.0 > > > Attached is the documentation for the crypto functions that are being added > to Drill 1.11.0. > https://gist.github.com/cgivre/63b25bdc85159bec484f069406858adc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-5658) Documentation for Drill Crypto Functions
[ https://issues.apache.org/jira/browse/DRILL-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens resolved DRILL-5658. --- Resolution: Fixed Document is posted here: https://drill.apache.org/docs/cryptography-functions/ > Documentation for Drill Crypto Functions > > > Key: DRILL-5658 > URL: https://issues.apache.org/jira/browse/DRILL-5658 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Charles Givre >Assignee: Bridget Bevens >Priority: Major > Labels: doc-impacting > Fix For: 1.16.0 > > > Attached is the documentation for the crypto functions that are being added > to Drill 1.11.0. > https://gist.github.com/cgivre/63b25bdc85159bec484f069406858adc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5658) Documentation for Drill Crypto Functions
[ https://issues.apache.org/jira/browse/DRILL-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-5658: -- Labels: doc-complete (was: doc-impacting) > Documentation for Drill Crypto Functions > > > Key: DRILL-5658 > URL: https://issues.apache.org/jira/browse/DRILL-5658 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Charles Givre >Assignee: Bridget Bevens >Priority: Major > Labels: doc-complete > Fix For: 1.16.0 > > > Attached is the documentation for the crypto functions that are being added > to Drill 1.11.0. > https://gist.github.com/cgivre/63b25bdc85159bec484f069406858adc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7014) Format plugin for LTSV files
[
https://issues.apache.org/jira/browse/DRILL-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7014:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Format plugin for LTSV files
>
>
> Key: DRILL-7014
> URL: https://issues.apache.org/jira/browse/DRILL-7014
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.15.0
>Reporter: Takako Shimamoto
>Assignee: Takako Shimamoto
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> I would like to contribute [this
> plugin|https://github.com/bizreach/drill-ltsv-plugin] to Drill.
> h4. Abstract
> storage-plugins-override.conf
> {code:json}
> "storage":{
> dfs: {
> type: "file",
> connection: "file:///",
> formats: {
> "ltsv": {
> "type": "ltsv",
> "extensions": [
> "ltsv"
> ]
> }
> },
> enabled: true
> }
> }
> {code}
> sample.ltsv
> {code}
> time:30/Nov/2016:00:55:08 +0900 host:xxx.xxx.xxx.xxx forwardedfor:- req:GET
> /v1/xxx HTTP/1.1 status:200 size:4968 referer:- ua:Java/1.8.0_131
> reqtime:2.532 apptime:2.532 vhost:api.example.com
> time:30/Nov/2016:00:56:37 +0900 host:xxx.xxx.xxx.xxx forwardedfor:- req:GET
> /v1/yyy HTTP/1.1 status:200 size:412 referer:- ua:Java/1.8.0_201
> reqtime:3.580 apptime:3.580 vhost:api.example.com
> {code}
> Run query
> {code:sh}
> root@1805183e9b65:/apache-drill-1.15.0# ./bin/drill-embedded
> Apache Drill 1.15.0
> "Drill must go on."
> 0: jdbc:drill:zk=local> SELECT * FROM
> dfs.`/apache-drill-1.15.0/sample-data/sample.ltsv` WHERE reqtime > 3.0;
> +-+--+---+---+-+---+--+-+--+--+--+
> |time | host | forwardedfor |
> req | status | size | referer | ua| reqtime |
> apptime | vhost |
> +-+--+---+---+-+---+--+-+--+--+--+
> | 30/Nov/2016:00:56:37 +0900 | xxx.xxx.xxx.xxx | - | GET
> /v1/yyy HTTP/1.1 | 200 | 412 | -| Java/1.8.0_201 | 3.580|
> 3.580| api.example.com |
> +-+--+---+---+-+---+--+-+--+--+--+
> 1 row selected (6.074 seconds)
> 0: jdbc:drill:zk=local>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7014) Format plugin for LTSV files
[
https://issues.apache.org/jira/browse/DRILL-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814738#comment-16814738
]
Bridget Bevens commented on DRILL-7014:
---
Hi [~shimamoto]
I've added the doc here: https://drill.apache.org/docs/ltsv-format-plugin/
Let me know if I need to change anything.
Thank you!
~Bridget
> Format plugin for LTSV files
>
>
> Key: DRILL-7014
> URL: https://issues.apache.org/jira/browse/DRILL-7014
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.15.0
>Reporter: Takako Shimamoto
>Assignee: Takako Shimamoto
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> I would like to contribute [this
> plugin|https://github.com/bizreach/drill-ltsv-plugin] to Drill.
> h4. Abstract
> storage-plugins-override.conf
> {code:json}
> "storage":{
> dfs: {
> type: "file",
> connection: "file:///",
> formats: {
> "ltsv": {
> "type": "ltsv",
> "extensions": [
> "ltsv"
> ]
> }
> },
> enabled: true
> }
> }
> {code}
> sample.ltsv
> {code}
> time:30/Nov/2016:00:55:08 +0900 host:xxx.xxx.xxx.xxx forwardedfor:- req:GET
> /v1/xxx HTTP/1.1 status:200 size:4968 referer:- ua:Java/1.8.0_131
> reqtime:2.532 apptime:2.532 vhost:api.example.com
> time:30/Nov/2016:00:56:37 +0900 host:xxx.xxx.xxx.xxx forwardedfor:- req:GET
> /v1/yyy HTTP/1.1 status:200 size:412 referer:- ua:Java/1.8.0_201
> reqtime:3.580 apptime:3.580 vhost:api.example.com
> {code}
> Run query
> {code:sh}
> root@1805183e9b65:/apache-drill-1.15.0# ./bin/drill-embedded
> Apache Drill 1.15.0
> "Drill must go on."
> 0: jdbc:drill:zk=local> SELECT * FROM
> dfs.`/apache-drill-1.15.0/sample-data/sample.ltsv` WHERE reqtime > 3.0;
> +-+--+---+---+-+---+--+-+--+--+--+
> |time | host | forwardedfor |
> req | status | size | referer | ua| reqtime |
> apptime | vhost |
> +-+--+---+---+-+---+--+-+--+--+--+
> | 30/Nov/2016:00:56:37 +0900 | xxx.xxx.xxx.xxx | - | GET
> /v1/yyy HTTP/1.1 | 200 | 412 | -| Java/1.8.0_201 | 3.580|
> 3.580| api.example.com |
> +-+--+---+---+-+---+--+-+--+--+--+
> 1 row selected (6.074 seconds)
> 0: jdbc:drill:zk=local>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7077) Add Function to Facilitate Time Series Analysis
[
https://issues.apache.org/jira/browse/DRILL-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7077:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Add Function to Facilitate Time Series Analysis
> ---
>
> Key: DRILL-7077
> URL: https://issues.apache.org/jira/browse/DRILL-7077
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> When analyzing time based data, you will often have to aggregate by time
> grains. While some time grains will be easy to calculate, others, such as
> quarter, can be quite difficult. These functions enable a user to quickly and
> easily aggregate data by various units of time. Usage is as follows:
> {code:java}
> SELECT
> FROM
> GROUP BY nearestDate(, {code}
> So let's say that a user wanted to count the number of hits on a web server
> per 15 minute, the query might look like this:
> {code:java}
> SELECT nearestDate(`eventDate`, '15MINUTE' ) AS eventDate,
> COUNT(*) AS hitCount
> FROM dfs.`log.httpd`
> GROUP BY nearestDate(`eventDate`, '15MINUTE'){code}
> Currently supports the following time units:
> * YEAR
> * QUARTER
> * MONTH
> * WEEK_SUNDAY
> * WEEK_MONDAY
> * DAY
> * HOUR
> * HALF_HOUR / 30MIN
> * QUARTER_HOUR / 15MIN
> * MINUTE
> * 30SECOND
> * 15SECOND
> * SECOND
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7077) Add Function to Facilitate Time Series Analysis
[
https://issues.apache.org/jira/browse/DRILL-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813987#comment-16813987
]
Bridget Bevens commented on DRILL-7077:
---
[~cgivre] the info is posted here:
https://drill.apache.org/docs/date-time-functions-and-arithmetic/#nearestdate
Let me know if I need to change anything.
Thanks,
Bridget
> Add Function to Facilitate Time Series Analysis
> ---
>
> Key: DRILL-7077
> URL: https://issues.apache.org/jira/browse/DRILL-7077
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> When analyzing time based data, you will often have to aggregate by time
> grains. While some time grains will be easy to calculate, others, such as
> quarter, can be quite difficult. These functions enable a user to quickly and
> easily aggregate data by various units of time. Usage is as follows:
> {code:java}
> SELECT
> FROM
> GROUP BY nearestDate(, {code}
> So let's say that a user wanted to count the number of hits on a web server
> per 15 minute, the query might look like this:
> {code:java}
> SELECT nearestDate(`eventDate`, '15MINUTE' ) AS eventDate,
> COUNT(*) AS hitCount
> FROM dfs.`log.httpd`
> GROUP BY nearestDate(`eventDate`, '15MINUTE'){code}
> Currently supports the following time units:
> * YEAR
> * QUARTER
> * MONTH
> * WEEK_SUNDAY
> * WEEK_MONDAY
> * DAY
> * HOUR
> * HALF_HOUR / 30MIN
> * QUARTER_HOUR / 15MIN
> * MINUTE
> * 30SECOND
> * 15SECOND
> * SECOND
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7077) Add Function to Facilitate Time Series Analysis
[
https://issues.apache.org/jira/browse/DRILL-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813895#comment-16813895
]
Bridget Bevens commented on DRILL-7077:
---
Hi [~cgivre],
I'm trying this function and may be doing something wrong, but 15SECOND and
30SECOND are not working for me:
select nearestdate(CAST(COLUMNS[2] as timestamp), '30SECOND') as nearest_second
from dfs.samples.`/bee/time.csv`;
Error: SYSTEM ERROR: DrillRuntimeException: [30SECOND] is not a valid time
statement. Expecting: [YEAR, QUARTER, MONTH, WEEK_SUNDAY, WEEK_MONDAY, DAY,
HOUR, HALF_HOUR, QUARTER_HOUR, MINUTE, HALF_MINUTE, QUARTER_MINUTE, SECOND]
Fragment 0:0
Please, refer to logs for more information.
[Error Id: f119202e-ec24-4670-83c2-14b4a7f83ebf on doc23.lab:31010]
(state=,code=0)
apache drill> select nearestdate(CAST(COLUMNS[2] as timestamp), 'SECOND') as
nearest_second from dfs.samples.`/bee/time.csv`;
+---+
|nearest_second |
+---+
| 2018-01-01 05:10:15.0 |
| 2017-02-02 01:02:03.0 |
| 2003-04-06 07:11:11.0 |
+---+
3 rows selected (0.191 seconds)
Are 15SECOND and 30SECOND supported?
Thanks,
Bridget
> Add Function to Facilitate Time Series Analysis
> ---
>
> Key: DRILL-7077
> URL: https://issues.apache.org/jira/browse/DRILL-7077
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> When analyzing time based data, you will often have to aggregate by time
> grains. While some time grains will be easy to calculate, others, such as
> quarter, can be quite difficult. These functions enable a user to quickly and
> easily aggregate data by various units of time. Usage is as follows:
> {code:java}
> SELECT
> FROM
> GROUP BY nearestDate(, {code}
> So let's say that a user wanted to count the number of hits on a web server
> per 15 minute, the query might look like this:
> {code:java}
> SELECT nearestDate(`eventDate`, '15MINUTE' ) AS eventDate,
> COUNT(*) AS hitCount
> FROM dfs.`log.httpd`
> GROUP BY nearestDate(`eventDate`, '15MINUTE'){code}
> Currently supports the following time units:
> * YEAR
> * QUARTER
> * MONTH
> * WEEK_SUNDAY
> * WEEK_MONDAY
> * DAY
> * HOUR
> * HALF_HOUR / 30MIN
> * QUARTER_HOUR / 15MIN
> * MINUTE
> * 30SECOND
> * 15SECOND
> * SECOND
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6582) SYSLOG (RFC-5424) Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-6582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812919#comment-16812919 ] Bridget Bevens commented on DRILL-6582: --- Hi [~cgivre], I've added the doc here: https://drill.apache.org/docs/syslog-format-plugin/ Let me know if I need to change anything. Thanks, Bridget > SYSLOG (RFC-5424) Format Plugin > --- > > Key: DRILL-6582 > URL: https://issues.apache.org/jira/browse/DRILL-6582 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: Future >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.16.0 > > > Many security log files are in the format defined by RFC-5424. A format > plugin which can read data formatted in according to this specification would > be very useful for security engineers as well as network engineers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6582) SYSLOG (RFC-5424) Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-6582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6582: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > SYSLOG (RFC-5424) Format Plugin > --- > > Key: DRILL-6582 > URL: https://issues.apache.org/jira/browse/DRILL-6582 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: Future >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > > Many security log files are in the format defined by RFC-5424. A format > plugin which can read data formatted in according to this specification would > be very useful for security engineers as well as network engineers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-540) Allow querying hive views in Drill
[
https://issues.apache.org/jira/browse/DRILL-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812894#comment-16812894
]
Bridget Bevens commented on DRILL-540:
--
Hi [~IhorHuzenko],
Is the following note okay?
_For storage-based authorization, access to Hive views depends on the user’s
permissions on the underlying tables in the view definition. When a user
selects from a Hive view, the view is expanded (converted into a query), and
the underlying tables referenced in the query are validated for permissions._
Thanks,
Bridget
> Allow querying hive views in Drill
> --
>
> Key: DRILL-540
> URL: https://issues.apache.org/jira/browse/DRILL-540
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Hive
>Reporter: Ramana Inukonda Nagaraj
>Assignee: Igor Guzenko
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently hive views cannot be queried from drill.
> This Jira aims to add support for Hive views in Drill.
> *Implementation details:*
> # Drill persists it's views metadata in file with suffix .view.drill using
> json format. For example:
> {noformat}
> {
> "name" : "view_from_calcite_1_4",
> "sql" : "SELECT * FROM `cp`.`store.json`WHERE `store_id` = 0",
> "fields" : [ {
> "name" : "*",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ "dfs", "tmp" ]
> }
> {noformat}
> Later Drill parses the metadata and uses it to treat view names in SQL as a
> subquery.
> 2. In Apache Hive metadata about views is stored in similar way to
> tables. Below is example from metastore.TBLS :
>
> {noformat}
> TBL_ID |CREATE_TIME |DB_ID |LAST_ACCESS_TIME |OWNER |RETENTION |SD_ID
> |TBL_NAME |TBL_TYPE |VIEW_EXPANDED_TEXT |
> ---||--|-|--|--|--|--|--|---|
> 2 |1542111078 |1 |0|mapr |0 |2 |cview
>|VIRTUAL_VIEW |SELECT COUNT(*) FROM `default`.`customers` |
> {noformat}
> 3. So in Hive metastore views are considered as tables of special type.
> And main benefit is that we also have expanded SQL definition of views (just
> like in view.drill files). Also reading of the metadata is already
> implemented in Drill with help of thrift Metastore API.
> 4. To enable querying of Hive views we'll reuse existing code for Drill
> views as much as possible. First in *_HiveSchemaFactory.getDrillTable_* for
> _*HiveReadEntry*_ we'll convert the metadata to instance of _*View*_ (_which
> is actually model for data persisted in .view.drill files_) and then based on
> this instance return new _*DrillViewTable*_. Using this approach drill will
> handle hive views the same way as if it was initially defined in Drill and
> persisted in .view.drill file.
> 5. For conversion of Hive types: from _*FieldSchema*_ to _*RelDataType*_
> we'll reuse existing code from _*DrillHiveTable*_, so the conversion
> functionality will be extracted and used for both (table and view) fields
> type conversions.
> *Security implications*
> Consider simple example case where we have users,
> {code:java}
> user0 user1 user2
>\ /
> group12
> {code}
> and sample db where object names contains user or group who should access
> them
> {code:java}
> db_all
> tbl_user0
> vw_user0
> tbl_group12
> vw_group12
> {code}
> There are two Hive authorization modes supported by Drill - SQL Standart and
> Strorage Based authorization. For SQL Standart authorization permissions
> were granted using SQL:
> {code:java}
> SET ROLE admin;
> GRANT SELECT ON db_all.tbl_user0 TO USER user0;
> GRANT SELECT ON db_all.vw_user0 TO USER user0;
> CREATE ROLE group12;
> GRANT ROLE group12 TO USER user1;
> GRANT ROLE group12 TO USER user2;
> GRANT SELECT ON db_all.tbl_group12 TO ROLE group12;
> GRANT SELECT ON db_all.vw_group12 TO ROLE group12;
> {code}
> And for Storage based authorization permissions were granted using commands:
> {code:java}
> hadoop fs -chown user0:user0 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 700 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 750 /user/hive/warehouse/db_all.db/tbl_group12
> hadoop fs -chown user1:group12
> /user/hive/warehouse/db_all.db/tbl_group12{code}
> Then the following table shows us results of queries for both authorization
> models.
>
> *SQL Standart | Storage Based
> Authorization*
> ||SQL||user0||user1||user2|| ||user0||user1||user2||
> |*Queries executed using Drill :*| | | | | | | |
> |SHOW TABLES IN
[jira] [Comment Edited] (DRILL-6985) Fix sqlline.bat issues on Windows and add drill-embedded.bat
[
https://issues.apache.org/jira/browse/DRILL-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812737#comment-16812737
]
Bridget Bevens edited comment on DRILL-6985 at 4/8/19 9:01 PM:
---
Hi [~vvysotskyi],
I've updated https://drill.apache.org/docs/installing-drill-on-windows/
and https://drill.apache.org/docs/starting-drill-on-windows/ with this
information.
Please let me know if I need to make any other changes.
Thanks,
Bridget
was (Author: bbevens):
Hi [~vvysotskyi],
I've updated https://drill.apache.org/docs/installing-drill-on-windows/
with this information.
Please let me know if I need to make any other changes.
Thanks,
Bridget
> Fix sqlline.bat issues on Windows and add drill-embedded.bat
>
>
> Key: DRILL-6985
> URL: https://issues.apache.org/jira/browse/DRILL-6985
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
> Environment: Windows 10
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> *For documentation*
> {{drill-embedded.bat}} was added as handy script to Start Drill on Windows
> without passing any params.
> Please updated the following section:
> https://drill.apache.org/docs/starting-drill-on-windows/
> Other issues covered in this Jira:
> {{sqlline.bat}} fails for the next cases:
> 1. Specified file in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f /tmp/q.sql
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 2. Specified file path that contains spaces:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f "/tmp/q q.sql"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> q.sql""=="test" was unexpected at this time.
> {noformat}
> 3. Specified query in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -e "select *
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> * was unexpected at this time.
> {noformat}
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -q "select 'a'
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> 'a' was unexpected at this time.
> {noformat}
> 4. Specified custom config location:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config=/tmp/conf
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 5. Specified custom config location with spaces in the path:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config="/tmp/conf test"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> test"" was unexpected at this time.
> {noformat}
> 6. Sqlline was run from non-bin directory:
> {noformat}
> apache-drill-1.15.0>bin\sqlline.bat -u "jdbc:drill:zk=local"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6985) Fix sqlline.bat issues on Windows and add drill-embedded.bat
[
https://issues.apache.org/jira/browse/DRILL-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812737#comment-16812737
]
Bridget Bevens commented on DRILL-6985:
---
Hi [~vvysotskyi],
I've updated https://drill.apache.org/docs/installing-drill-on-windows/
with this information.
Please let me know if I need to make any other changes.
Thanks,
Bridget
> Fix sqlline.bat issues on Windows and add drill-embedded.bat
>
>
> Key: DRILL-6985
> URL: https://issues.apache.org/jira/browse/DRILL-6985
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
> Environment: Windows 10
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> *For documentation*
> {{drill-embedded.bat}} was added as handy script to Start Drill on Windows
> without passing any params.
> Please updated the following section:
> https://drill.apache.org/docs/starting-drill-on-windows/
> Other issues covered in this Jira:
> {{sqlline.bat}} fails for the next cases:
> 1. Specified file in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f /tmp/q.sql
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 2. Specified file path that contains spaces:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f "/tmp/q q.sql"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> q.sql""=="test" was unexpected at this time.
> {noformat}
> 3. Specified query in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -e "select *
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> * was unexpected at this time.
> {noformat}
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -q "select 'a'
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> 'a' was unexpected at this time.
> {noformat}
> 4. Specified custom config location:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config=/tmp/conf
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 5. Specified custom config location with spaces in the path:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config="/tmp/conf test"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> test"" was unexpected at this time.
> {noformat}
> 6. Sqlline was run from non-bin directory:
> {noformat}
> apache-drill-1.15.0>bin\sqlline.bat -u "jdbc:drill:zk=local"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6985) Fix sqlline.bat issues on Windows and add drill-embedded.bat
[
https://issues.apache.org/jira/browse/DRILL-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6985:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Fix sqlline.bat issues on Windows and add drill-embedded.bat
>
>
> Key: DRILL-6985
> URL: https://issues.apache.org/jira/browse/DRILL-6985
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
> Environment: Windows 10
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> *For documentation*
> {{drill-embedded.bat}} was added as handy script to Start Drill on Windows
> without passing any params.
> Please updated the following section:
> https://drill.apache.org/docs/starting-drill-on-windows/
> Other issues covered in this Jira:
> {{sqlline.bat}} fails for the next cases:
> 1. Specified file in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f /tmp/q.sql
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 2. Specified file path that contains spaces:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -f "/tmp/q q.sql"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> q.sql""=="test" was unexpected at this time.
> {noformat}
> 3. Specified query in the argument:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -e "select *
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> * was unexpected at this time.
> {noformat}
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local" -q "select 'a'
> from sys.version"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> 'a' was unexpected at this time.
> {noformat}
> 4. Specified custom config location:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config=/tmp/conf
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
> 5. Specified custom config location with spaces in the path:
> {noformat}
> apache-drill-1.15.0\bin>sqlline.bat -u "jdbc:drill:zk=local"
> --config="/tmp/conf test"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> test"" was unexpected at this time.
> {noformat}
> 6. Sqlline was run from non-bin directory:
> {noformat}
> apache-drill-1.15.0>bin\sqlline.bat -u "jdbc:drill:zk=local"
> DRILL_ARGS - " -u jdbc:drill:zk=local"
> HADOOP_HOME not detected...
> HBASE_HOME not detected...
> Calculating Drill classpath...
> Error: Could not find or load main class sqlline.SqlLine
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5679) Document JAVA_HOME requirements for installing Drill in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812734#comment-16812734
]
Bridget Bevens commented on DRILL-5679:
---
Hi [~arina],
I've updated https://drill.apache.org/docs/installing-drill-on-windows/ with
the information here.
Was this JIRA meant for Drill in distributed mode on Windows, embedded mode on
Windows, both?
Currently, we do not have a Drill in distributed mode on Windows doc, but I can
add a note in the prereqs section for Drill in distributed mode.
Thanks,
Bridget
> Document JAVA_HOME requirements for installing Drill in distributed mode
>
>
> Key: DRILL-5679
> URL: https://issues.apache.org/jira/browse/DRILL-5679
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Bridget Bevens
>Priority: Major
> Labels: doc-complete
> Fix For: Future
>
>
> There is general requirement that JAVA_HOME variable should not contain
> spaces.
> For example, during Drill installation in distributed mode on Windows user
> can see the following error:
> {noformat}
> C:\Drill/bin/runbit: line 107: exec: C:\Program: not found
> {noformat}
> There are two options to fix this problem:
> {noformat}
> 1. Install JAVA in directory without spaces.
> 2. Replace "Program Files" in your JAVA_HOME variable to progra~1 or progra~2
> (if in x86).
> Example: JAVA_HOME="C:\progra~1\Java\jdk1.7.0_71"
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5679) Document JAVA_HOME requirements for installing Drill in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-5679:
--
Labels: doc-complete (was: doc-impacting)
> Document JAVA_HOME requirements for installing Drill in distributed mode
>
>
> Key: DRILL-5679
> URL: https://issues.apache.org/jira/browse/DRILL-5679
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Bridget Bevens
>Priority: Major
> Labels: doc-complete
> Fix For: Future
>
>
> There is general requirement that JAVA_HOME variable should not contain
> spaces.
> For example, during Drill installation in distributed mode on Windows user
> can see the following error:
> {noformat}
> C:\Drill/bin/runbit: line 107: exec: C:\Program: not found
> {noformat}
> There are two options to fix this problem:
> {noformat}
> 1. Install JAVA in directory without spaces.
> 2. Replace "Program Files" in your JAVA_HOME variable to progra~1 or progra~2
> (if in x86).
> Example: JAVA_HOME="C:\progra~1\Java\jdk1.7.0_71"
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-540) Allow querying hive views in Drill
[
https://issues.apache.org/jira/browse/DRILL-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808132#comment-16808132
]
Bridget Bevens commented on DRILL-540:
--
Hi [~IhorHuzenko] and [~vitalii],
Aside from removing the note stating hive views are not supported, do I need to
add any other information to the docs, for example, shouold I aslo include the
warning?
Warning: Because views in Hive aren't present as physical files and access
can't be granted using file system commands, then access to Hive views for
Storage Based Authorization is based on the underlying tables used in view
definition. For current example views were defined as selection over
appropriate tables.
Thanks,
Bridget
> Allow querying hive views in Drill
> --
>
> Key: DRILL-540
> URL: https://issues.apache.org/jira/browse/DRILL-540
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Hive
>Reporter: Ramana Inukonda Nagaraj
>Assignee: Igor Guzenko
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently hive views cannot be queried from drill.
> This Jira aims to add support for Hive views in Drill.
> *Implementation details:*
> # Drill persists it's views metadata in file with suffix .view.drill using
> json format. For example:
> {noformat}
> {
> "name" : "view_from_calcite_1_4",
> "sql" : "SELECT * FROM `cp`.`store.json`WHERE `store_id` = 0",
> "fields" : [ {
> "name" : "*",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ "dfs", "tmp" ]
> }
> {noformat}
> Later Drill parses the metadata and uses it to treat view names in SQL as a
> subquery.
> 2. In Apache Hive metadata about views is stored in similar way to
> tables. Below is example from metastore.TBLS :
>
> {noformat}
> TBL_ID |CREATE_TIME |DB_ID |LAST_ACCESS_TIME |OWNER |RETENTION |SD_ID
> |TBL_NAME |TBL_TYPE |VIEW_EXPANDED_TEXT |
> ---||--|-|--|--|--|--|--|---|
> 2 |1542111078 |1 |0|mapr |0 |2 |cview
>|VIRTUAL_VIEW |SELECT COUNT(*) FROM `default`.`customers` |
> {noformat}
> 3. So in Hive metastore views are considered as tables of special type.
> And main benefit is that we also have expanded SQL definition of views (just
> like in view.drill files). Also reading of the metadata is already
> implemented in Drill with help of thrift Metastore API.
> 4. To enable querying of Hive views we'll reuse existing code for Drill
> views as much as possible. First in *_HiveSchemaFactory.getDrillTable_* for
> _*HiveReadEntry*_ we'll convert the metadata to instance of _*View*_ (_which
> is actually model for data persisted in .view.drill files_) and then based on
> this instance return new _*DrillViewTable*_. Using this approach drill will
> handle hive views the same way as if it was initially defined in Drill and
> persisted in .view.drill file.
> 5. For conversion of Hive types: from _*FieldSchema*_ to _*RelDataType*_
> we'll reuse existing code from _*DrillHiveTable*_, so the conversion
> functionality will be extracted and used for both (table and view) fields
> type conversions.
> *Security implications*
> Consider simple example case where we have users,
> {code:java}
> user0 user1 user2
>\ /
> group12
> {code}
> and sample db where object names contains user or group who should access
> them
> {code:java}
> db_all
> tbl_user0
> vw_user0
> tbl_group12
> vw_group12
> {code}
> There are two Hive authorization modes supported by Drill - SQL Standart and
> Strorage Based authorization. For SQL Standart authorization permissions
> were granted using SQL:
> {code:java}
> SET ROLE admin;
> GRANT SELECT ON db_all.tbl_user0 TO USER user0;
> GRANT SELECT ON db_all.vw_user0 TO USER user0;
> CREATE ROLE group12;
> GRANT ROLE group12 TO USER user1;
> GRANT ROLE group12 TO USER user2;
> GRANT SELECT ON db_all.tbl_group12 TO ROLE group12;
> GRANT SELECT ON db_all.vw_group12 TO ROLE group12;
> {code}
> And for Storage based authorization permissions were granted using commands:
> {code:java}
> hadoop fs -chown user0:user0 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 700 /user/hive/warehouse/db_all.db/tbl_user0
> hadoop fs -chmod 750 /user/hive/warehouse/db_all.db/tbl_group12
> hadoop fs -chown user1:group12
> /user/hive/warehouse/db_all.db/tbl_group12{code}
> Then the following table shows us results of queries for both authorization
> models.
> *SQL
> Standart Storage Based
[jira] [Comment Edited] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805567#comment-16805567
]
Bridget Bevens edited comment on DRILL-7038 at 4/1/19 9:49 PM:
---
Thanks, [~KazydubB]!
Is this okay to add to the document?
Starting in Drill 1.16, Drill uses a Value operator instead of a Scan operator
to read data when a query references a directory column only and has a DISTINCT
or GROUP BY operation. Instead of scanning all directory columns, Drill either
reads the specified column from the metadata cache file (if one exists) or
Drill selects directly from the directory (partition location). The presence of
the Values operator (instead of the Scan operator) in the query plan indicates
that Drill is using this optimization, as shown in the following examples:
select distinct dir0 from `/logs`;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select distinct dir0 from `/logs`;
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
select dir0 from `/logs` group by dir0;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select dir0 from `/logs` group by dir0;
>
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
Thanks,
Bridget
was (Author: bbevens):
Thanks, [~KazydubB]!
Is this okay to add to the document?
Starting in 1.16, Drill uses a Values operator instead of the Scan operator for
DISTINCT and GROUP BY queries on tables and directories, as shown in the
following examples:
select distinct dir0 from `/logs`;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select distinct dir0 from `/logs`;
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
select dir0 from `/logs` group by dir0;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select dir0 from `/logs` group by dir0;
>
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
Thanks,
Bridget
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL:
[jira] [Commented] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805567#comment-16805567
]
Bridget Bevens commented on DRILL-7038:
---
Thanks, [~KazydubB]!
Is this okay to add to the document?
Starting in 1.16, Drill uses a Values operator instead of the Scan operator for
DISTINCT and GROUP BY queries on tables and directories, as shown in the
following examples:
select distinct dir0 from `/logs`;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select distinct dir0 from `/logs`;
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
select dir0 from `/logs` group by dir0;
+--+
| dir0 |
+--+
| 2015 |
| 2016 |
| 2017 |
+--+
explain plan for select dir0 from `/logs` group by dir0;
>
+--+--+
| text
| json
|
+--+--+
| 00-00Screen
00-01 Project(dir0=[$0])
00-02StreamAgg(group=[{0}])
00-03 Sort(sort0=[$0], dir0=[ASC])
00-04Values(tuples=[[{ '2015' }, { '2015' }, { '2016' }, { '2015'
}, { '2017' }, { '2015' }]])
Thanks,
Bridget
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL: https://issues.apache.org/jira/browse/DRILL-7038
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> For tables with hive-style partitions like
> {code}
> /table/2018/Q1
> /table/2018/Q2
> /table/2019/Q1
> etc.
> {code}
> if any of the following queries is run:
> {code}
> select distinct dir0 from dfs.`/table`
> {code}
> {code}
> select dir0 from dfs.`/table` group by dir0
> {code}
> it will actually scan every single record in the table rather than just
> getting a list of directories at the dir0 level. This applies even when
> cached metadata is available. This is a big penalty especially as the
> datasets grow.
> To avoid such situations, a logical prune rule can be used to collect
> partition columns (`dir0`), either from metadata cache (if available) or
> group scan, and drop unnecessary files from being read. The rule will be
> applied on following conditions:
> 1) all queried columns are partitoin columns, and
> 2) either {{DISTINCT}} or {{GROUP BY}} operations are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803526#comment-16803526
]
Bridget Bevens edited comment on DRILL-7038 at 3/28/19 1:50 AM:
Hi [~KazydubB],
I'm planning to add the following information to this page:
https://drill.apache.org/docs/querying-directories/
Can you please review and let me know if the information is accurate?
Starting in Drill 1.16, if a query on a file system table references
partitioned columns only and also has a DISTINCT or GROUP BY operation, Drill
only reads the data partition(s) referenced. For example, if a table, table1,
is partitioned into the following directories:
/table1/2016
/table1/2017
/table1/2018
And you run either of the following queries against table1:
select distinct dir0 from dfs.`/table`;
select dir0 from dfs.`/table` group by dir0;
Drill only scans dir0 (/table1/2016) instead of scanning all three of the
directories.
Thanks,
Bridget
was (Author: bbevens):
Hi [~KazydubB],
I'm planning to add the following information to this page:
https://drill.apache.org/docs/querying-directories/
Can you please review and let me know if the information is accurate?
Starting in Drill 1.16, if a query on a file system table references
partitioned columns only and also has a DISTINCT or GROUP BY operation, Drill
only reads the data partition(s) referenced. For example, if a table, table1,
is partitioned into the following directories:
{{/table1/2016
/table1/2017
/table1/2018}}
And you run either of the following queries against table1:
{{select distinct dir0 from dfs.`/table`;
select dir0 from dfs.`/table` group by dir0;}}
Drill only scans dir0 (/table1/2016) instead of scanning all three of the
directories.
Thanks,
Bridget
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL: https://issues.apache.org/jira/browse/DRILL-7038
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> For tables with hive-style partitions like
> {code}
> /table/2018/Q1
> /table/2018/Q2
> /table/2019/Q1
> etc.
> {code}
> if any of the following queries is run:
> {code}
> select distinct dir0 from dfs.`/table`
> {code}
> {code}
> select dir0 from dfs.`/table` group by dir0
> {code}
> it will actually scan every single record in the table rather than just
> getting a list of directories at the dir0 level. This applies even when
> cached metadata is available. This is a big penalty especially as the
> datasets grow.
> To avoid such situations, a logical prune rule can be used to collect
> partition columns (`dir0`), either from metadata cache (if available) or
> group scan, and drop unnecessary files from being read. The rule will be
> applied on following conditions:
> 1) all queried columns are partitoin columns, and
> 2) either {{DISTINCT}} or {{GROUP BY}} operations are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7038) Queries on partitioned columns scan the entire datasets
[
https://issues.apache.org/jira/browse/DRILL-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803526#comment-16803526
]
Bridget Bevens commented on DRILL-7038:
---
Hi [~KazydubB],
I'm planning to add the following information to this page:
https://drill.apache.org/docs/querying-directories/
Can you please review and let me know if the information is accurate?
Starting in Drill 1.16, if a query on a file system table references
partitioned columns only and also has a DISTINCT or GROUP BY operation, Drill
only reads the data partition(s) referenced. For example, if a table, table1,
is partitioned into the following directories:
{{/table1/2016
/table1/2017
/table1/2018}}
And you run either of the following queries against table1:
{{select distinct dir0 from dfs.`/table`;
select dir0 from dfs.`/table` group by dir0;}}
Drill only scans dir0 (/table1/2016) instead of scanning all three of the
directories.
Thanks,
Bridget
> Queries on partitioned columns scan the entire datasets
> ---
>
> Key: DRILL-7038
> URL: https://issues.apache.org/jira/browse/DRILL-7038
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> For tables with hive-style partitions like
> {code}
> /table/2018/Q1
> /table/2018/Q2
> /table/2019/Q1
> etc.
> {code}
> if any of the following queries is run:
> {code}
> select distinct dir0 from dfs.`/table`
> {code}
> {code}
> select dir0 from dfs.`/table` group by dir0
> {code}
> it will actually scan every single record in the table rather than just
> getting a list of directories at the dir0 level. This applies even when
> cached metadata is available. This is a big penalty especially as the
> datasets grow.
> To avoid such situations, a logical prune rule can be used to collect
> partition columns (`dir0`), either from metadata cache (if available) or
> group scan, and drop unnecessary files from being read. The rule will be
> applied on following conditions:
> 1) all queried columns are partitoin columns, and
> 2) either {{DISTINCT}} or {{GROUP BY}} operations are performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7085) Drill Statistics: analyze table cmd fails
[
https://issues.apache.org/jira/browse/DRILL-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7085:
--
Description:
On the latest build we are seeing in the regression results that the analyze
cmd is failing.
This is the dir:
{code:java}
[root@qa101-108 bin]# hadoop fs -ls /drill/testdat03-05 18:26
/drill/testdata/a/table_stats/data_with_0 Found 1 items -rwxr-xr-x 3 root root
487 2019-table_stats/data_with_0/0_0_0.parquet
{code}
{code:java}
0: jdbc:drill:drillbit=10.10.101.108> set `planner.statistics.use`=true;
+---+--+
| ok | summary |
+---+--+
| true | planner.statistics.use updated. |
+---+--+
1 row selected (0.164 seconds)
{code}
{code:java}
0: jdbc:drill:drillbit=10.10.101.108> analyze table `table_stats/data_with_0`
compute statistics;
++--+
| ok | summary
|
++--+
| false | Table table_stats/data_with_0 is not supported by ANALYZE. Support
is currently limited to directory-based Parquet tables. |
++--+
1 row selected (0.901 seconds)
{code}
was:
On the latest build we are seeing in the regression results that the analyze
cmd is failing.
This is the dir:
{code:java}
[root@qa101-108 bin]# hadoop fs -ls /drill/testdata/table_stats/data_with_0
Found 1 items -rwxr-xr-x 3 root root 487 2019-03-05 18:26
/drill/testdata/table_stats/data_with_0/0_0_0.parquet
{code}
{code:java}
0: jdbc:drill:drillbit=10.10.101.108> set `planner.statistics.use`=true;
+---+--+
| ok | summary |
+---+--+
| true | planner.statistics.use updated. |
+---+--+
1 row selected (0.164 seconds)
{code}
{code:java}
0: jdbc:drill:drillbit=10.10.101.108> analyze table `table_stats/data_with_0`
compute statistics;
++--+
| ok | summary
|
++--+
| false | Table table_stats/data_with_0 is not supported by ANALYZE. Support
is currently limited to directory-based Parquet tables. |
++--+
1 row selected (0.901 seconds)
{code}
> Drill Statistics: analyze table cmd fails
> -
>
> Key: DRILL-7085
> URL: https://issues.apache.org/jira/browse/DRILL-7085
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Anisha Reddy
>Assignee: Gautam Parai
>Priority: Blocker
> Fix For: 1.16.0
>
>
> On the latest build we are seeing in the regression results that the analyze
> cmd is failing.
> This is the dir:
> {code:java}
> [root@qa101-108 bin]# hadoop fs -ls /drill/testdat03-05 18:26
> /drill/testdata/a/table_stats/data_with_0 Found 1 items -rwxr-xr-x 3 root
> root 487 2019-table_stats/data_with_0/0_0_0.parquet
> {code}
> {code:java}
> 0: jdbc:drill:drillbit=10.10.101.108> set `planner.statistics.use`=true;
> +---+--+
> | ok | summary |
> +---+--+
> | true | planner.statistics.use updated. |
> +---+--+
> 1 row selected (0.164 seconds)
> {code}
> {code:java}
> 0: jdbc:drill:drillbit=10.10.101.108> analyze table `table_stats/data_with_0`
> compute statistics;
> ++--+
> | ok | summary
> |
> ++--+
> | false | Table table_stats/data_with_0 is not supported by ANALYZE. Support
> is currently limited to directory-based Parquet tables. |
> ++--+
> 1 row selected (0.901 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-1328) Support table statistics
[ https://issues.apache.org/jira/browse/DRILL-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-1328: -- Labels: doc-impacting (was: ) > Support table statistics > > > Key: DRILL-1328 > URL: https://issues.apache.org/jira/browse/DRILL-1328 > Project: Apache Drill > Issue Type: Improvement >Reporter: Cliff Buchanan >Assignee: Gautam Parai >Priority: Major > Labels: doc-impacting > Fix For: 1.16.0 > > Attachments: 0001-PRE-Set-value-count-in-splitAndTransfer.patch > > > This consists of several subtasks > * implement operators to generate statistics > * add "analyze table" support to parser/planner > * create a metadata provider to allow statistics to be used by optiq in > planning optimization > * implement statistics functions > Right now, the bulk of this functionality is implemented, but it hasn't been > rigorously tested and needs to have some definite answers for some of the > parts "around the edges" (how analyze table figures out where the table > statistics are located, how a table "append" should work in a read only file > system) > Also, here are a few known caveats: > * table statistics are collected by creating a sql query based on the string > path of the table. This should probably be done with a Table reference. > * Case sensitivity for column statistics is probably iffy > * Math for combining two column NDVs into a joint NDV should be checked. > * Schema changes aren't really being considered yet. > * adding getDrillTable is probably unnecessary; it might be better to do > getTable().unwrap(DrillTable.class) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7071: -- Fix Version/s: (was: Future) 1.16.0 > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Priority: Minor > Labels: doc, documentation, keyword, reserved-word > Fix For: 1.16.0 > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens reassigned DRILL-7071: - Assignee: Bridget Bevens > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Assignee: Bridget Bevens >Priority: Minor > Labels: doc, documentation, keyword, reserved-word > Fix For: 1.16.0 > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7071) Reserved words documentation udpate
[ https://issues.apache.org/jira/browse/DRILL-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-7071: -- Labels: doc documentation keyword reserved-word (was: doc docuentation keyword reserved-word) > Reserved words documentation udpate > --- > > Key: DRILL-7071 > URL: https://issues.apache.org/jira/browse/DRILL-7071 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Vitalii Diravka >Priority: Minor > Labels: doc, documentation, keyword, reserved-word > Fix For: Future > > > Last time a lot of reserved keywords were added to Drill project, for > instance in DRILL-1328 or DRILL-7058 will introduce new one too. Therefore > Drill reserved keywords in documentation should be updated: > [https://drill.apache.org/docs/reserved-keywords/] > These words should be obtained from these sections of Drill Parser file: > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L30] > and > [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/data/Parser.tdd#L390] > _Note:_ this list will be updated after the next Calcite version update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4587) Document Drillbit launch options
[ https://issues.apache.org/jira/browse/DRILL-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-4587: -- Fix Version/s: (was: 1.16.0) 1.17.0 > Document Drillbit launch options > > > Key: DRILL-4587 > URL: https://issues.apache.org/jira/browse/DRILL-4587 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Reporter: Paul Rogers >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.17.0 > > > Drill provides the drillbit.sh script to launch Drill. When Drill is run in > production environments, or when managed by a tool such as Mesos or YARN, > customers have many options to customize the launch options. We should > document this information as below. > The user can configure Drill launch in one of four ways, depending on their > needs. > 1. Using the properties in drill-override.conf. Sets only startup and runtime > properties. All drillbits should use a copy of the file so that properties > set here apply to all drill bits and to client applications. > 2. By setting environment variables prior to launching Drill. See the list > below. Use this to customize properties per drill-bit, such as for setting > port numbers. This option is useful when launching Drill from a tool such as > Mesos or YARN. > 3. By setting environment variables in $DRILL_HOME/conf/drill-env.sh. See the > list below. This script is intended to be unique to each node and is another > way to customize properties for this one node. > 4. In Drill 1.7 and later, the administrator can set Drill configuration > options directly on the launch command as shown below. This option is also > useful when launching Drill from a tool such as YARN or Mesos. Options are of > the form: > $ drillbit.sh start -Dvariable=value > For example, to control the HTTP port: > $ drillbit.sh start -Ddrill.exec.http.port=8099 > Properties are of three types. > 1. Launch-only properties: those that can be set only through environment > variables (such as JAVA_HOME.) > 2. Drill startup properties which can be set in the locations detailed below. > 3. Drill runtime properties which are set in drill-override.conf also via SQL. > Drill startup propeties can be set in a number of locations. Those listed > later take precedence over those listed earlier. > 1. Drill-override.conf as identified by DRILL_CONF_DIR or its default. > 2. Set in the environment using DRILL_JAVA_OPTS or DRILL_DRILLBIT_JAVA_OPTS. > 3. Set in drill-env.sh using the above two variables. > 4. Set on the drill.bit command line as explained above. (Drill 1.7 and > later.) > You can see the actual set of properties used (from items 2-3 above) by using > the "debug" command (Drill 1.7 or later): > $ drillbit.sh debug -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7015) Improve documentation for PARTITION BY
[
https://issues.apache.org/jira/browse/DRILL-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7015:
--
Fix Version/s: (was: 1.16.0)
1.17.0
> Improve documentation for PARTITION BY
> --
>
> Key: DRILL-7015
> URL: https://issues.apache.org/jira/browse/DRILL-7015
> Project: Apache Drill
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.15.0
>Reporter: Boaz Ben-Zvi
>Assignee: Bridget Bevens
>Priority: Minor
> Fix For: 1.17.0
>
>
> The documentation for CREATE TABLE AS (CTAS) shows the syntax of the command,
> without the optional PARTITION BY clause. That option is only mentioned later
> under the usage notes.
> *+_Suggestion_+*: Add this optional clause to the syntax (same as for CREATE
> TEMPORARY TABLE (CTTAS)). And mention that this option is only applicable
> when storing in Parquet.
> And the documentation for CREATE TEMPORARY TABLE (CTTAS), the comment says:
> {panel}
> An optional parameter that can *only* be used to create temporary tables with
> the Parquet data format.
> {panel}
> Which can mistakenly be understood as "only for temporary tables".
> *_+Suggestion+_*: erase the "to create temporary tables" part (not needed, as
> it is implied from the context of this page).
> *_+Last suggestion+_*: In the documentation for the PARTITION BY clause, can
> add an example using the implicit column "filename" to demonstrate how the
> partitioning column puts each distinct value into a separate file. For
> example, add in the "Other Examples" section :
> {noformat}
> 0: jdbc:drill:zk=local> select distinct r_regionkey, filename from mytable1;
> +--++
> | r_regionkey |filename|
> +--++
> | 2| 0_0_3.parquet |
> | 1| 0_0_2.parquet |
> | 0| 0_0_1.parquet |
> | 3| 0_0_4.parquet |
> | 4| 0_0_5.parquet |
> +--++
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6945) Update INFORMATION_SCHEMA.SCHEMATA table description
[
https://issues.apache.org/jira/browse/DRILL-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6945:
--
Fix Version/s: (was: 1.16.0)
1.17.0
> Update INFORMATION_SCHEMA.SCHEMATA table description
>
>
> Key: DRILL-6945
> URL: https://issues.apache.org/jira/browse/DRILL-6945
> Project: Apache Drill
> Issue Type: Task
> Components: Documentation
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Bridget Bevens
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> https://drill.apache.org/docs/querying-the-information-schema/
> Currently documentation states that SCHEMA table contains only several
> columns:
> {noformat}
> The SCHEMATA table contains the CATALOG_NAME and SCHEMA_NAME columns. To
> allow maximum flexibility inside BI tools, the only catalog that Drill
> supports is DRILL.
> {noformat}
> In reality it contains far more columns (especially TYPE and IS_MUTABLE)
> which can be considered to be documented:
> {noformat}
> drill (information_schema)>select * from schemata;
> +---+--+---++-+
> | CATALOG_NAME | SCHEMA_NAME | SCHEMA_OWNER | TYPE |
> IS_MUTABLE |
> +---+--+---++-+
> | DRILL | cp.default || file | NO
> |
> | DRILL | dfs.default || file | NO
> |
> | DRILL | dfs.myschemainitcap || file | YES
> |
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6728) DRILL-4864 - doc udfs for date, time, timestamp functions
[ https://issues.apache.org/jira/browse/DRILL-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6728: -- Fix Version/s: (was: 1.16.0) 1.17.0 > DRILL-4864 - doc udfs for date, time, timestamp functions > - > > Key: DRILL-6728 > URL: https://issues.apache.org/jira/browse/DRILL-6728 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Reporter: Bridget Bevens >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5046) Add documentation for directory based partition pruning
[ https://issues.apache.org/jira/browse/DRILL-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-5046: -- Fix Version/s: (was: 1.16.0) 1.17.0 > Add documentation for directory based partition pruning > --- > > Key: DRILL-5046 > URL: https://issues.apache.org/jira/browse/DRILL-5046 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Rahul Challapalli >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.17.0 > > > Drill's documentation for partition pruning should cover the below 2 features > 1. Directory based partition pruning > 2. Partition pruning based on auto-partitioned parquet files > The first one seems to be missing from our documentation. At the very least > we should cover > a. How we can leverage this feature to avoid full table scans > b. How this feature works in-conjunction with metadata cache pruning > c. A few examples which involve using wildcards for one of the sub-directories -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6794) Document the JDBC properties required to retrieve result sets in batches while querying large tables
[ https://issues.apache.org/jira/browse/DRILL-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6794: -- Fix Version/s: (was: 1.16.0) 1.17.0 > Document the JDBC properties required to retrieve result sets in batches > while querying large tables > > > Key: DRILL-6794 > URL: https://issues.apache.org/jira/browse/DRILL-6794 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.14.0 >Reporter: Rahul Raj >Assignee: Bridget Bevens >Priority: Major > Labels: doc-impacting > Fix For: 1.17.0 > > > Document the JDBC properties required to retrieve result sets in batches > while querying large tables > Querying large tables using JDBC plugin causes OOM as most JDBC drivers cache > the entire result set at the client by default. > To avoid this additional parameters needs to be specified with the JDBC > connection string so that the driver fetches records in batches and reloads > when exhausted. > For postgres driver set autocommit mode to false - > jdbc:postgresql://url:port/schema?defaultAutoCommit=false > Links > [1] https://issues.apache.org/jira/browse/DRILL-4177 > [2] https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example > [3] https://www.postgresql.org/docs/9.3/static/ecpg-sql-set-autocommit.html > [4] https://jdbc.postgresql.org/documentation/head/ds-cpds.htm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6911) Documentation issue - Hadoop core-site.xml is not supported by Drill to read S3 credentials
[
https://issues.apache.org/jira/browse/DRILL-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756663#comment-16756663
]
Bridget Bevens commented on DRILL-6911:
---
Thanks, [~denysord88]. I removed the note.
Best,
Bridget
> Documentation issue - Hadoop core-site.xml is not supported by Drill to read
> S3 credentials
> ---
>
> Key: DRILL-6911
> URL: https://issues.apache.org/jira/browse/DRILL-6911
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Bridget Bevens
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.16.0
>
>
> In the Drill S3 documentation https://drill.apache.org/docs/s3-storage-plugin/
> Section "Providing AWS Credentials" describing 3 ways to setup AWS S3
> credentials in Drill:
> - storage plugin;
> - Drill-specific core-site.xml;
> - existing S3 configuration for Hadoop.
> Third item is not supported by Drill. Hadoop core-site.xml config file may
> contains S3 credentials, but Drill doesn't read any S3 parameters directly
> from Hadoop config file.
> Third item
> {code:java}
> In a Hadoop environment, you can use the existing S3 configuration for
> Hadoop. The AWS credentials should already be defined. All you need to do is
> configure the S3 storage plugin.
> {code}
> should be removed from the document
> https://drill.apache.org/docs/s3-storage-plugin/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6911) Documentation issue - Hadoop core-site.xml is not supported by Drill to read S3 credentials
[
https://issues.apache.org/jira/browse/DRILL-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6911:
--
Labels: doc-complete (was: doc-impacting)
> Documentation issue - Hadoop core-site.xml is not supported by Drill to read
> S3 credentials
> ---
>
> Key: DRILL-6911
> URL: https://issues.apache.org/jira/browse/DRILL-6911
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Denys Ordynskiy
>Assignee: Bridget Bevens
>Priority: Major
> Labels: doc-complete
> Fix For: 1.16.0
>
>
> In the Drill S3 documentation https://drill.apache.org/docs/s3-storage-plugin/
> Section "Providing AWS Credentials" describing 3 ways to setup AWS S3
> credentials in Drill:
> - storage plugin;
> - Drill-specific core-site.xml;
> - existing S3 configuration for Hadoop.
> Third item is not supported by Drill. Hadoop core-site.xml config file may
> contains S3 credentials, but Drill doesn't read any S3 parameters directly
> from Hadoop config file.
> Third item
> {code:java}
> In a Hadoop environment, you can use the existing S3 configuration for
> Hadoop. The AWS credentials should already be defined. All you need to do is
> configure the S3 storage plugin.
> {code}
> should be removed from the document
> https://drill.apache.org/docs/s3-storage-plugin/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7001) Documentation - renaming columns name in csv header
[
https://issues.apache.org/jira/browse/DRILL-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-7001:
--
Labels: doc-impacting (was: )
> Documentation - renaming columns name in csv header
> ---
>
> Key: DRILL-7001
> URL: https://issues.apache.org/jira/browse/DRILL-7001
> Project: Apache Drill
> Issue Type: Wish
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Minor
> Labels: doc-impacting
>
> Don't know how if this is the best place for this request but,
> Some operation are realized that eventually change the name of the column
> when requesting a csvh file (with header),
> These operations are not documented.
> Although it's possible to read
> [HeaderBuilder.java|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/HeaderBuilder.java],
> It will be interesting to create a section in documentation to explain at
> least the principle of these different cases to avoid stupid
> problems/difficulties
> List of operations (maybe not exhaustive) :
> * Trim() on CSV column name
> {noformat}
> Name , Age,PoB , Info
> =>
> `Name`, `Age`, `PoB` and `Info`{noformat}
> * Others characters than [a-zA-Z0-9_] are replace by '_' (underscore)
> {noformat}
> Name,Sum$,em@il
> =>
> `Name`,'`Sum_`,`em_il`{noformat}
> * Fieldname starting with '_' (underscore) are prefixed by 'col'
> {noformat}
> _name,_age_,pob_,_col_
> =>
> `col_name`, `col_age_`, `pob_`, `col_col_`{noformat}
> * Fieldname starting with [^a-zA-Z] are prefixed 'col_'
> {noformat}
> 0_name, 1_age,@pob,#other1,'other2'
> =>
> `col_0_name`, `col_1_age`, `col_pob`, `col_other1`, `col_other2_`{noformat}
> * Quotation marks are removed
> * If char is unique
> ** if [a-zA-Z] do nothing
> ** elif [0-9] prefix with col_
> ** else reanme in column_[0-9]+ where [0-9]+ designs the position of the
> column
> * Duplicate columns names (case insensitive) are suffixed with _[0-9]+
> (starting from "_2")
> {noformat}
> 0_name,col_0_name,colx,COLX,colx,colx_2
> =>
> `col_0_name`, `col_0_name_2`, `colx`, `COLX_2`, `colx_3`, `colx_2_2`{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6982) Affected rows count is not returned by Drill if return_result_set_for_ddl is false
[
https://issues.apache.org/jira/browse/DRILL-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749392#comment-16749392
]
Bridget Bevens commented on DRILL-6982:
---
Hi [~kkhatua],
Based on https://issues.apache.org/jira/browse/DRILL-6834, Drill doc was
reviewed and approved. The Drill documentation states the following about the
the exec.return_result_set_for_ddl option:
_“When set to false, Drill returns the affected row count, and the result set
is null.”_
So, based on this JIRA, seems that Drill is not having the expected behavior.
I’m guessing this JIRA is asking for Drill to return the affected rows count
for DDL statements when the exec.return_result_set_for_ddl option is set to
false.
If you set the option to false and run the following create table command:
CREATE TABLE tmp.`name_key2` (N_NAME, N_NATIONKEY) AS SELECT N_NATIONKEY,
N_NAME FROM
dfs.`/root/drill-1.15/apache-drill-1.15.0-SNAPSHOT/sample-data/nation.parquet`;
Drill creates the table, but returns the following instead of the affected row
count:
No rows affected (2.714 seconds)
Thanks,
Bridget
> Affected rows count is not returned by Drill if return_result_set_for_ddl is
> false
> --
>
> Key: DRILL-6982
> URL: https://issues.apache.org/jira/browse/DRILL-6982
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Anton Gozhiy
>Priority: Major
>
> *Prerequisites:*
> {code:sql}
> set `exec.query.return_result_set_for_ddl`= false;
> {code}
> *Query:*
> {code:sql}
> create table dfs.tmp.`nation as select * from cp.`tpch/nation.parquet`;
> {code}
> *Expected result:*
> Drill should return the number of affected rows (25 in this case)
> *Actual Result:*
> The table was created, but affected rows count wasn't returned:
> {noformat}
> No rows affected (1.755 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-4721) Doc "dateDiff" in Drill
[
https://issues.apache.org/jira/browse/DRILL-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens resolved DRILL-4721.
---
Resolution: Fixed
Reviewer: Vitalii Diravka
Fix Version/s: 1.15.0
Documented the date_diff function.
> Doc "dateDiff" in Drill
> ---
>
> Key: DRILL-4721
> URL: https://issues.apache.org/jira/browse/DRILL-4721
> Project: Apache Drill
> Issue Type: Task
> Components: Documentation
>Reporter: Bridget Bevens
>Assignee: Bridget Bevens
>Priority: Minor
> Fix For: 1.15.0
>
>
> Fwd: [mapr-tech-qa:12242]Doc "dateDiff" in Drill
> Inbox
> x
> Neeraja Rentachintala
> 3:32 PM (19 minutes ago)
> to me
> we should document the datediff function. Cisco was asking about it today.
> -- Forwarded message --
> From: Bob Rumsby
> Date: Thu, Aug 20, 2015 at 11:56 AM
> Subject: Re: [mapr-tech-qa:12242] "dateDiff" in Drill
> To: "mapr-tech...@maprtech.com"
> Yes, it should be. We can fix that.
> Bob
> On Thu, Aug 20, 2015 at 11:46 AM, Joseph Blue wrote:
> Seems as though functions like datediff should be here:
> https://drill.apache.org/docs/date-time-functions-and-arithmetic/
> On Thu, Aug 20, 2015 at 11:32 AM, Ted Dunning wrote:
> Joe
> Do you have a suggestion for the docs? Perhaps a few cross links would make
> it better?
> Sent from my iPhone
> On Aug 20, 2015, at 5:36, Joseph Blue wrote:
> OK, duh. Thanks for that. I just went down the wrong path in the
> documentation.
> This obviously does exactly what I want.
> On Wed, Aug 19, 2015 at 10:26 PM, Mehant Baid wrote:
> You can use the datediff function as follows:
> select datediff(date '2008-2-23', date '2008-1-20') from cp.`employee.json`
> limit 1;
> +-+
> | EXPR$0 |
> +-+
> | 34 |
> +-+
> Thanks
> Mehant
> On Wed, Aug 19, 2015 at 7:00 PM, Joseph Blue wrote:
> Want to get the difference between two dates in Drill. I used the AGE
> function which produces an interval. But I want the answer in days, not
> months & days. Am I missing something from the documentation?
> select age(cast('2015-01-01' as timestamp),cast('2014-11-30' as timestamp))
> from sys.version;
> +-+
> | EXPR$0 |
> +-+
> | P1M2D |
> +-+
> --
> Joseph Blue
> Data Scientist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4721) Doc "dateDiff" in Drill
[
https://issues.apache.org/jira/browse/DRILL-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743459#comment-16743459
]
Bridget Bevens commented on DRILL-4721:
---
Hi [~vitalii],
I've added content for the date_diff function
[here|https://drill.apache.org/docs/date-time-functions-and-arithmetic/#date_diff].
Please have a look and let me know if I need to change anything.
Thanks,
Bridget
> Doc "dateDiff" in Drill
> ---
>
> Key: DRILL-4721
> URL: https://issues.apache.org/jira/browse/DRILL-4721
> Project: Apache Drill
> Issue Type: Task
> Components: Documentation
>Reporter: Bridget Bevens
>Assignee: Bridget Bevens
>Priority: Minor
>
> Fwd: [mapr-tech-qa:12242]Doc "dateDiff" in Drill
> Inbox
> x
> Neeraja Rentachintala
> 3:32 PM (19 minutes ago)
> to me
> we should document the datediff function. Cisco was asking about it today.
> -- Forwarded message --
> From: Bob Rumsby
> Date: Thu, Aug 20, 2015 at 11:56 AM
> Subject: Re: [mapr-tech-qa:12242] "dateDiff" in Drill
> To: "mapr-tech...@maprtech.com"
> Yes, it should be. We can fix that.
> Bob
> On Thu, Aug 20, 2015 at 11:46 AM, Joseph Blue wrote:
> Seems as though functions like datediff should be here:
> https://drill.apache.org/docs/date-time-functions-and-arithmetic/
> On Thu, Aug 20, 2015 at 11:32 AM, Ted Dunning wrote:
> Joe
> Do you have a suggestion for the docs? Perhaps a few cross links would make
> it better?
> Sent from my iPhone
> On Aug 20, 2015, at 5:36, Joseph Blue wrote:
> OK, duh. Thanks for that. I just went down the wrong path in the
> documentation.
> This obviously does exactly what I want.
> On Wed, Aug 19, 2015 at 10:26 PM, Mehant Baid wrote:
> You can use the datediff function as follows:
> select datediff(date '2008-2-23', date '2008-1-20') from cp.`employee.json`
> limit 1;
> +-+
> | EXPR$0 |
> +-+
> | 34 |
> +-+
> Thanks
> Mehant
> On Wed, Aug 19, 2015 at 7:00 PM, Joseph Blue wrote:
> Want to get the difference between two dates in Drill. I used the AGE
> function which produces an interval. But I want the answer in days, not
> months & days. Am I missing something from the documentation?
> select age(cast('2015-01-01' as timestamp),cast('2014-11-30' as timestamp))
> from sys.version;
> +-+
> | EXPR$0 |
> +-+
> | P1M2D |
> +-+
> --
> Joseph Blue
> Data Scientist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6922) QUERY-level options are shown on Profiles tab
[ https://issues.apache.org/jira/browse/DRILL-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736489#comment-16736489 ] Bridget Bevens commented on DRILL-6922: --- Hi [~arina], I'm not able to edit this JIRA, not sure why, but I would like to remove the doc-impacting label and replace it with doc-complete. Thanks, Bridget > QUERY-level options are shown on Profiles tab > - > > Key: DRILL-6922 > URL: https://issues.apache.org/jira/browse/DRILL-6922 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Bohdan Kazydub >Assignee: Arina Ielchiieva >Priority: Blocker > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Option `exec.return_result_set_for_ddl` is shown on Web UI's Profiles even > when it was not set explicitly. The issue is because the option is being set > on query level internally. > *For documentation* > Option `exec.return_result_set_for_ddl` was renamed to > `exec.query.return_result_set_for_ddl`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6922) QUERY-level options are shown on Profiles tab
[ https://issues.apache.org/jira/browse/DRILL-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736482#comment-16736482 ] Bridget Bevens commented on DRILL-6922: --- Changed all instances of exec.return_result_set_for_ddl to exec.query.return_result_set_for_ddl in documentation. Thanks, Bridget > QUERY-level options are shown on Profiles tab > - > > Key: DRILL-6922 > URL: https://issues.apache.org/jira/browse/DRILL-6922 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Bohdan Kazydub >Assignee: Arina Ielchiieva >Priority: Blocker > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Option `exec.return_result_set_for_ddl` is shown on Web UI's Profiles even > when it was not set explicitly. The issue is because the option is being set > on query level internally. > *For documentation* > Option `exec.return_result_set_for_ddl` was renamed to > `exec.query.return_result_set_for_ddl`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6922) QUERY-level options are shown on Profiles tab
[ https://issues.apache.org/jira/browse/DRILL-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736481#comment-16736481 ] Bridget Bevens commented on DRILL-6922: --- Changed all instances of exec.return_result_set_for_ddl to exec.query.return_result_set_for_ddl in the documentation. Let me know if there's any issue. Thanks, Bridget > QUERY-level options are shown on Profiles tab > - > > Key: DRILL-6922 > URL: https://issues.apache.org/jira/browse/DRILL-6922 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Bohdan Kazydub >Assignee: Arina Ielchiieva >Priority: Blocker > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Option `exec.return_result_set_for_ddl` is shown on Web UI's Profiles even > when it was not set explicitly. The issue is because the option is being set > on query level internally. > *For documentation* > Option `exec.return_result_set_for_ddl` was renamed to > `exec.query.return_result_set_for_ddl`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6853) Parquet Complex Reader for nested schema should have configurable memory or max records to fetch
[ https://issues.apache.org/jira/browse/DRILL-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-6853: -- Labels: doc-complete pull-request-available ready-to-commit (was: doc-impacting pull-request-available ready-to-commit) > Parquet Complex Reader for nested schema should have configurable memory or > max records to fetch > > > Key: DRILL-6853 > URL: https://issues.apache.org/jira/browse/DRILL-6853 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Nitin Sharma >Assignee: salim achouche >Priority: Major > Labels: doc-complete, pull-request-available, ready-to-commit > Fix For: 1.15.0 > > > Parquet Complex reader while fetching nested schema should have configurable > memory or max records to fetch and not default to 4000 records. > While scanning TB of data with wider columns, this could easily cause OOM > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6853) Parquet Complex Reader for nested schema should have configurable memory or max records to fetch
[ https://issues.apache.org/jira/browse/DRILL-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736457#comment-16736457 ] Bridget Bevens commented on DRILL-6853: --- Documented the new option here: https://drill.apache.org/docs/start-up-options/#configuring-start-up-options Please let me know if I need to make any changes to the description. Thanks, Bridget > Parquet Complex Reader for nested schema should have configurable memory or > max records to fetch > > > Key: DRILL-6853 > URL: https://issues.apache.org/jira/browse/DRILL-6853 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Nitin Sharma >Assignee: salim achouche >Priority: Major > Labels: doc-impacting, pull-request-available, ready-to-commit > Fix For: 1.15.0 > > > Parquet Complex reader while fetching nested schema should have configurable > memory or max records to fetch and not default to 4000 records. > While scanning TB of data with wider columns, this could easily cause OOM > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6941) Incorrect EARLY_LIMIT0_OPT_KEY description
[
https://issues.apache.org/jira/browse/DRILL-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736153#comment-16736153
]
Bridget Bevens commented on DRILL-6941:
---
I think the doc has the correct description:
https://drill.apache.org/docs/limit-clause/#limit-0-optimizations
Please let me know if I need to include any other info.
Thanks,
Bridget
> Incorrect EARLY_LIMIT0_OPT_KEY description
> --
>
> Key: DRILL-6941
> URL: https://issues.apache.org/jira/browse/DRILL-6941
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently in ExecConstants:
> {noformat}
> public static final String EARLY_LIMIT0_OPT_KEY =
> "planner.enable_limit0_optimization";
> public static final BooleanValidator EARLY_LIMIT0_OPT = new
> BooleanValidator(EARLY_LIMIT0_OPT_KEY,
> new OptionDescription("Sets the type of identifier quotes for the SQL
> parser. Default is backticks ('`'). The SQL parser accepts double quotes
> ('\"') and square brackets ('['). (Drill 1.11+)"));
> {noformat}
> Should be something like this:
> planner.enable_limit0_optimization
> {noformat}
> Enables the query planner to determine data types returned by a query during
> the planning phase before scanning data. Default is true. (Drill 1.9 and
> later)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6941) Incorrect EARLY_LIMIT0_OPT_KEY description
[
https://issues.apache.org/jira/browse/DRILL-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6941:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Incorrect EARLY_LIMIT0_OPT_KEY description
> --
>
> Key: DRILL-6941
> URL: https://issues.apache.org/jira/browse/DRILL-6941
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-complete, ready-to-commit
> Fix For: 1.16.0
>
>
> Currently in ExecConstants:
> {noformat}
> public static final String EARLY_LIMIT0_OPT_KEY =
> "planner.enable_limit0_optimization";
> public static final BooleanValidator EARLY_LIMIT0_OPT = new
> BooleanValidator(EARLY_LIMIT0_OPT_KEY,
> new OptionDescription("Sets the type of identifier quotes for the SQL
> parser. Default is backticks ('`'). The SQL parser accepts double quotes
> ('\"') and square brackets ('['). (Drill 1.11+)"));
> {noformat}
> Should be something like this:
> planner.enable_limit0_optimization
> {noformat}
> Enables the query planner to determine data types returned by a query during
> the planning phase before scanning data. Default is true. (Drill 1.9 and
> later)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6941) Incorrect EARLY_LIMIT0_OPT_KEY description
[
https://issues.apache.org/jira/browse/DRILL-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736136#comment-16736136
]
Bridget Bevens commented on DRILL-6941:
---
Hi [~kkhatua],
LGTM
> Incorrect EARLY_LIMIT0_OPT_KEY description
> --
>
> Key: DRILL-6941
> URL: https://issues.apache.org/jira/browse/DRILL-6941
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.16.0
>
>
> Currently in ExecConstants:
> {noformat}
> public static final String EARLY_LIMIT0_OPT_KEY =
> "planner.enable_limit0_optimization";
> public static final BooleanValidator EARLY_LIMIT0_OPT = new
> BooleanValidator(EARLY_LIMIT0_OPT_KEY,
> new OptionDescription("Sets the type of identifier quotes for the SQL
> parser. Default is backticks ('`'). The SQL parser accepts double quotes
> ('\"') and square brackets ('['). (Drill 1.11+)"));
> {noformat}
> Should be something like this:
> planner.enable_limit0_optimization
> {noformat}
> Enables the query planner to determine data types returned by a query during
> the planning phase before scanning data. Default is true. (Drill 1.9 and
> later)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733470#comment-16733470
]
Bridget Bevens commented on DRILL-3610:
---
Hi [~vvysotskyi],
No worries; I've updated the queries for TIMESTAMPADD|DIFF with CAST function.
Hope they are okay now.
Thanks,
Bridget
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730852#comment-16730852
]
Bridget Bevens commented on DRILL-3610:
---
Hi [~vvysotskyi],
I've updatd the descriptions and examples again. Hopefully they are accurate
now, but if not, please let me know and I'll continue to work on them. :-)
https://drill.apache.org/docs/date-time-functions-and-arithmetic/#timestampadd
https://drill.apache.org/docs/date-time-functions-and-arithmetic/#timestampdiff
Thanks,
Bridget
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-5735: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.15.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730492#comment-16730492 ] Bridget Bevens commented on DRILL-5735: --- hi [~kkhatua], I've added this section to the docs based on info in this JIRA: https://drill.apache.org/docs/planning-and-execution-options/#setting-options-from-the-drill-web-ui Let me know if I need to change anything. Thanks, Bridget > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.15.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-3933) Error execute select command line sqlline -u -q
[ https://issues.apache.org/jira/browse/DRILL-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bridget Bevens updated DRILL-3933: -- Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit) > Error execute select command line sqlline -u -q > --- > > Key: DRILL-3933 > URL: https://issues.apache.org/jira/browse/DRILL-3933 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jon >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.15.0 > > > I'm newbie with Drill and Jira, so sorry if this is not the correct site. > When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from > hive.database.table;' " return: > "select anaconda-ks.cfg build.out install.log install.log.syslog > ranger_tutorial sandbox.info start_ambari.sh start_hbase.sh start_solr.sh > stop_solr.sh from hive.database.table;" > Error: PARSE ERROR: Encountered "." at line 1, column 29. > Was expecting one of: > "FROM" ... > "," ... > So, to fix this, i should type all columns to do this one work. > But, if I used UI, in localhost:8047/query, the query works. Drill is > connected to Hive with plugin of course. Is this a bug or something? Bad > conf.? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3933) Error execute select command line sqlline -u -q
[ https://issues.apache.org/jira/browse/DRILL-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730488#comment-16730488 ] Bridget Bevens commented on DRILL-3933: --- Hi [~vitalii], Thanks for the info. I've updated the page with the following section: https://drill.apache.org/docs/configuring-the-drill-shell/#sqlline-connection-parameters Please have a look and let me know if I need to change anything. Thanks, Bridget > Error execute select command line sqlline -u -q > --- > > Key: DRILL-3933 > URL: https://issues.apache.org/jira/browse/DRILL-3933 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jon >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > I'm newbie with Drill and Jira, so sorry if this is not the correct site. > When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from > hive.database.table;' " return: > "select anaconda-ks.cfg build.out install.log install.log.syslog > ranger_tutorial sandbox.info start_ambari.sh start_hbase.sh start_solr.sh > stop_solr.sh from hive.database.table;" > Error: PARSE ERROR: Encountered "." at line 1, column 29. > Was expecting one of: > "FROM" ... > "," ... > So, to fix this, i should type all columns to do this one work. > But, if I used UI, in localhost:8047/query, the query works. Drill is > connected to Hive with plugin of course. Is this a bug or something? Bad > conf.? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3610) TimestampAdd/Diff (SQL_TSI_) functions
[
https://issues.apache.org/jira/browse/DRILL-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730078#comment-16730078
]
Bridget Bevens commented on DRILL-3610:
---
Hi [~vvysotskyi],
Thanks for reviewing. I updated the pages based on your feedback; hopefully
they are correct now. If not, let me know and I'll change them again.
https://drill.apache.org/docs/date-time-functions-and-arithmetic/#timestampadd
https://drill.apache.org/docs/date-time-functions-and-arithmetic/#timestampdiff
Thanks,
Bridget
> TimestampAdd/Diff (SQL_TSI_) functions
> --
>
> Key: DRILL-3610
> URL: https://issues.apache.org/jira/browse/DRILL-3610
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Reporter: Andries Engelbrecht
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> Add TimestampAdd and TimestampDiff (SQL_TSI) functions for year, quarter,
> month, week, day, hour, minute, second.
> Examples
> SELECT CAST(TIMESTAMPADD(SQL_TSI_QUARTER,1,Date('2013-03-31'), SQL_DATE) AS
> `column_quarter`
> FROM `table_in`
> HAVING (COUNT(1) > 0)
> SELECT `table_in`.`datetime` AS `column1`,
> `table`.`Key` AS `column_Key`,
> TIMESTAMPDIFF(SQL_TSI_MINUTE,to_timestamp('2004-07-04',
> '-MM-dd'),`table_in`.`datetime`) AS `sum_datediff_minute`
> FROM `calcs`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3933) Error execute select command line sqlline -u -q
[ https://issues.apache.org/jira/browse/DRILL-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730039#comment-16730039 ] Bridget Bevens commented on DRILL-3933: --- Hi [~arina] and [~vitalii], I see this JIRA is marked with doc-impact, but I'm not sure what needs to be documented. In the description, the sqlline connection string includes a query with -q paramenter. Is this what needs to be documented - how to connect via sqlline and include the -q parameter? Thanks, Bridget > Error execute select command line sqlline -u -q > --- > > Key: DRILL-3933 > URL: https://issues.apache.org/jira/browse/DRILL-3933 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jon >Assignee: Arina Ielchiieva >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > I'm newbie with Drill and Jira, so sorry if this is not the correct site. > When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from > hive.database.table;' " return: > "select anaconda-ks.cfg build.out install.log install.log.syslog > ranger_tutorial sandbox.info start_ambari.sh start_hbase.sh start_solr.sh > stop_solr.sh from hive.database.table;" > Error: PARSE ERROR: Encountered "." at line 1, column 29. > Was expecting one of: > "FROM" ... > "," ... > So, to fix this, i should type all columns to do this one work. > But, if I used UI, in localhost:8047/query, the query works. Drill is > connected to Hive with plugin of course. Is this a bug or something? Bad > conf.? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6611:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6611) Add [meta]-[Enter] js handler for query form submission
[
https://issues.apache.org/jira/browse/DRILL-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729976#comment-16729976
]
Bridget Bevens commented on DRILL-6611:
---
Hi [~kkhatua],
I've included information for this jira on this page:
https://drill.apache.org/docs/starting-the-web-ui/
I will also include this new feature in the release notes.
Let me know if I need to change anything.
Thanks,
Bridget
> Add [meta]-[Enter] js handler for query form submission
> ---
>
> Key: DRILL-6611
> URL: https://issues.apache.org/jira/browse/DRILL-6611
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Bob Rudis
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> The new ACE-based SQL query editor is great. Being able to submit the form
> without using a mouse would be even better.
> Adding:
>
> {noformat}
> document.getElementById('queryForm')
> .addEventListener('keydown', function(e) {
> if (!(e.keyCode == 13 && e.metaKey)) return;
> if (e.target.form) doSubmitQueryWithUserName();
> });
> {noformat}
> {{to ./exec/java-exec/src/main/resources/rest/query/query.ftl adds such
> support.}}
> I can file a PR with the code if desired.
> --
> Functionality (for the documentation):
> This JIRA's commit introduces the following to Drill:
> When composing queries in the web query editor it is now possible to submit
> the query text by using the {{Meta+Enter}} key combination. This will trigger
> the same action as pressing the {{Submit}} button. On Mac keyboards
> {{Meta+Enter}} is {{Cmd+Enter}}. On Windows or Linux is {{Ctrl+Enter}} though
> Linux users may have keymapped the {{Meta}} key to another physical keyboard
> key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6668) In Web Console, highlight options that are different from default values
[
https://issues.apache.org/jira/browse/DRILL-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6668:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> In Web Console, highlight options that are different from default values
>
>
> Key: DRILL-6668
> URL: https://issues.apache.org/jira/browse/DRILL-6668
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Paul Rogers
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
> Attachments: screenshot-1.png
>
>
> Suppose you inherit a Drill setup created by someone else (or by you, some
> time in the past). Or, suppose you are a support person. You want to know
> which Drill options have been changed from the defaults.
> The Web UI conveniently displays all options. But, there is no indication of
> which might have non-default values.
> After the improvements of the last year, the information needed to detect
> non-default values is now available. Would be great to mark these values.
> Perhaps using colors, perhaps with words.
> For example:
> *planner.width.max_per_node* 200 \[Update]
> Or
> planner.width.max_per_node (system) 200 \[Update]
> (The Web UI does not, I believe, show session settings, since the Web UI has
> no sessions. I believe the custom values are all set by {{ALTER SYSTEM}}.
> Otherwise, we could also have a "(session)" suffix above.)
> Then, in addition to the {{[Update]}} button, for non default values, also
> provide a {{[Reset]}} button that does the same as {{ALTER SESSION RESET}}.
> planner.width.max_per_node (session) 200 \[Update] \[Reset]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6668) In Web Console, highlight options that are different from default values
[
https://issues.apache.org/jira/browse/DRILL-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729971#comment-16729971
]
Bridget Bevens commented on DRILL-6668:
---
Hi [~kkhatua],
I've updated the following page with info about the new options, including
examples:
https://drill.apache.org/docs/planning-and-execution-options/#setting-options-from-the-drill-web-ui
I will also mention this feature in the release notes.
Let me know if I need to make any changes to the doc content.
Thanks,
Bridget
> In Web Console, highlight options that are different from default values
>
>
> Key: DRILL-6668
> URL: https://issues.apache.org/jira/browse/DRILL-6668
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Paul Rogers
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
> Attachments: screenshot-1.png
>
>
> Suppose you inherit a Drill setup created by someone else (or by you, some
> time in the past). Or, suppose you are a support person. You want to know
> which Drill options have been changed from the defaults.
> The Web UI conveniently displays all options. But, there is no indication of
> which might have non-default values.
> After the improvements of the last year, the information needed to detect
> non-default values is now available. Would be great to mark these values.
> Perhaps using colors, perhaps with words.
> For example:
> *planner.width.max_per_node* 200 \[Update]
> Or
> planner.width.max_per_node (system) 200 \[Update]
> (The Web UI does not, I believe, show session settings, since the Web UI has
> no sessions. I believe the custom values are all set by {{ALTER SYSTEM}}.
> Otherwise, we could also have a "(session)" suffix above.)
> Then, in addition to the {{[Update]}} button, for non default values, also
> provide a {{[Reset]}} button that does the same as {{ALTER SESSION RESET}}.
> planner.width.max_per_node (session) 200 \[Update] \[Reset]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6544) Allow timestamp / date / time formatting when displaying on Web UI
[
https://issues.apache.org/jira/browse/DRILL-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bridget Bevens updated DRILL-6544:
--
Labels: doc-complete ready-to-commit (was: doc-impacting ready-to-commit)
> Allow timestamp / date / time formatting when displaying on Web UI
> --
>
> Key: DRILL-6544
> URL: https://issues.apache.org/jira/browse/DRILL-6544
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Anton Gozhiy
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: doc-complete, ready-to-commit
> Fix For: 1.15.0
>
>
> *Query:*
> {code:sql}
> select timestamp '2008-2-23 12:23:34' from (values(1));
> {code}
> *Expected result (from sqline):*
> 2008-02-23 12:23:34.0
> *Actual result (from Drill UI):*
> 2008-02-23T12:23:34
> *For documentation*
> Three new session options are added:
> web.display_format.timestamp
> web.display_format.time
> web.display_format.date
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6544) Allow timestamp / date / time formatting when displaying on Web UI
[
https://issues.apache.org/jira/browse/DRILL-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729968#comment-16729968
]
Bridget Bevens commented on DRILL-6544:
---
Hi [~angozhiy],
I've updated the following page with info about the new options, including
examples:
https://drill.apache.org/docs/planning-and-execution-options/#setting-options-from-the-drill-web-ui
I will also mention these new options in the release notes.
Let me know if I need to make any changes to the doc content.
Thanks,
Bridget
> Allow timestamp / date / time formatting when displaying on Web UI
> --
>
> Key: DRILL-6544
> URL: https://issues.apache.org/jira/browse/DRILL-6544
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Anton Gozhiy
>Assignee: Anton Gozhiy
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
>
> *Query:*
> {code:sql}
> select timestamp '2008-2-23 12:23:34' from (values(1));
> {code}
> *Expected result (from sqline):*
> 2008-02-23 12:23:34.0
> *Actual result (from Drill UI):*
> 2008-02-23T12:23:34
> *For documentation*
> Three new session options are added:
> web.display_format.timestamp
> web.display_format.time
> web.display_format.date
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6934) Update the option documentation for planner.enable_unnest_lateral
[ https://issues.apache.org/jira/browse/DRILL-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729864#comment-16729864 ] Bridget Bevens edited comment on DRILL-6934 at 12/27/18 9:14 PM: - [~shamirwasia] I see, this is in the Drill Web UI. The option description is: Enables lateral join functionality. Default is false. (Drill 1.14+) Need to change to Default is true. Drill 1.15+ was (Author: bbevens): Hi [~shamirwasia], Can you please include a link to the page that you are referring to, as I cannot find a reference to it being disabled by default. https://drill.apache.org/docs/lateral-join/ is updated for 1.15 stating that it is enabled by default. You may need to refresh the page to see the update. In the Drill Web UI, I see that the option is true by default. Thanks, Bridget > Update the option documentation for planner.enable_unnest_lateral > - > > Key: DRILL-6934 > URL: https://issues.apache.org/jira/browse/DRILL-6934 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.16.0 > > > The planner option documentation on WebUI shows that default value of option > planner.enable_unnest_lateral is false which is wrong. It is true by default > post from 1.15 onwards and hence needs to be changed to reflect that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
