subject:"\[GitHub\] \[drill\] paul\-rogers commented on a change in pull request #2030\: Update docs for Metastore to point that all format plugins are supported"

[GitHub] [drill] paul-rogers commented on a change in pull request #2030: Update docs for Metastore to point that all format plugins are supported

2020-03-20 Thread GitBox

paul-rogers commented on a change in pull request #2030: Update docs for 
Metastore to point that all format plugins are supported
URL: https://github.com/apache/drill/pull/2030#discussion_r395773086
 
 

 ##
 File path: 
_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##
 @@ -250,6 +282,9 @@ A table can be divided into directories, called 
"partitions". The `PARTITIONS` t
  - Applies to tables stored as Parquet files and only when stored in the `DFS` 
storage plugin.
  - Disabled by default. You must enable this feature through the 
`metastore.enabled` system/session option.
 
+### Limitations of the 1.18 release
+ - Applies to all file system storage plugin formats except for MaprDB.
 
 Review comment:
   Again, since Drill 1.17 is released, should we list Drill 1.17 limitations 
as well?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2030: Update docs for Metastore to point that all format plugins are supported

2020-03-20 Thread GitBox

paul-rogers commented on a change in pull request #2030: Update docs for
Metastore to point that all format plugins are supported
URL: https://github.com/apache/drill/pull/2030#discussion_r395772674

##
File path:
_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
##
@@ -103,20 +105,50 @@ Schema information and summary statistics also computed
and stored for table seg

The detailed metadata schema is described
[here](https://github.com/apache/drill/tree/master/metastore/metastore-api#metastore-tables).
You can try out the metadata to get a sense of what is available, by using the
- [Inspect the Metastore using `INFORMATION_SCHEMA`
tables]({{site.baseurl}}/docs/using-drill-metastore/#inspect-the-metastore-using-information_schema-tables)
tutorial.
+ [Inspect the Metastore using `INFORMATION_SCHEMA`
tables](#inspect-the-metastore-using-information_schema-tables) tutorial.

Every table described by the Metastore may be a bare file or one or more files
that reside in one or more directories.

If a table consists of a single directory or file, then it is non-partitioned.
The single directory can contain any number of files.
Larger tables tend to have subdirectories. Each subdirectory is a partition
and such a table are called "partitioned".
-Please refer to [Exposing Drill Metastore metadata through
`INFORMATION_SCHEMA`
tables]({{site.baseurl}}/docs/using-drill-metastore/#exposing-drill-metastore-metadata-through-information_schema-tables)
+Please refer to [Exposing Drill Metastore metadata through
`INFORMATION_SCHEMA`
tables](#exposing-drill-metastore-metadata-through-information_schema-tables)
for information, how to query partitions and segments metadata.

A traditional database divides tables into schemas and tables.
Drill can connect to any number of data sources, each of which may have its
own schema.
As a result, the Metastore labels tables with a combination of (plugin
configuration name, workspace name, table name).
Note that if before renaming any of these items, you must delete table's
Metadata entry and recreate it after renaming.

+### Using schema provisioning feature with Drill Metastore
+
+The Drill Metastore holds both schema and statistics information for a table.
The `ANALYZE` command can infer the table
+ schema for well-defined tables (such as many Parquet tables). Some tables are
too complex or variable for Drill's
+ schema inference to work well. For example, JSON tables often omit fields or
have long runs of nulls so that Drill
+ cannot determine column types. In these cases, you can specify the correct
schema based on your knowledge of the
+ table's structure. You specify a schema in the `ANALYZE` command using the
+ [Schema
provisioning]({{site.baseurl}}/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
syntax.
+
+Please refer to [Provisioning schema for Drill
Metastore](#provisioning-schema-for-drill-metastore) for examples of usage.
+
+### Schema priority
+
+Drill allows the following ways for providing table schema:
+ - providing schema with table function:
+ - specifying inline schema;
+ - specifying path to the schema file;
+ - using schema file in table root directory;
+ - using schema from Drill Metastore.
+
+The highest priority has schema provided in table function.
+
+Second priority has schema file (if `store.table.use_schema_file` is enabled).
+
+If neither of the above schema sources wasn't specified, schema from Drill
Metastore will be used.
+
+Regardless of the source of the schema, it will be used and handled in the
same way.
+
+Table metadata from Drill Metastore will be used if it is available regardless
of the schema source.
+

Review comment:
Drill uses metadata during both query planning and execution. Drill gives
you multiple ways to provide a schema.

When you run the `ANALYZE TABLE` command, Drill will uses the following
rules for the table schema to be stored in the Metastore. In priority order:

* A schema file, created with `CREATE OR REPLACE SCHEMA`, in the table root
directory.
* Schema inferred from file data.

To plan a query, Drill requires information about your file partitions (if
any) and about row and column cardinality. Drill does not use the provided
schema for planning as it does not provide this metadata. Instead, at plan time
Drill obtains metadata from one of the following, again in priority order:

* The Drill Metastore, if available.
* Inferred from file data. Drill scans the table's directory structure to
identify partitions. Drill estimates row counts based on the file size. Drill
uses default estimates for column cardinality.

At query execution time, a schema tells Drill the shape of your data and how
that data should be converted to Drill's SQL types. Your choices for
execution-time schema, in priority order, are:

* With a table function:
- specify

[GitHub] [drill] paul-rogers commented on a change in pull request #2030: Update docs for Metastore to point that all format plugins are supported

2020-03-20 Thread GitBox

paul-rogers commented on a change in pull request #2030: Update docs for 
Metastore to point that all format plugins are supported
URL: https://github.com/apache/drill/pull/2030#discussion_r395758027
 
 

 ##
 File path: 
_docs/performance-tuning/drill-metastore/010-using-drill-metastore.md
 ##
 @@ -1,14 +1,16 @@
 ---
 title: "Using Drill Metastore"
 parent: "Drill Metastore"
-date: 2020-03-03
+date: 2020-03-17
 ---
 
 Drill 1.17 introduces the Drill Metastore which stores the table schema and 
table statistics. Statistics allow Drill to better create optimal query plans.
 
 The Metastore is a Beta feature; it is subject to change. We encourage you to 
try it and provide feedback.
 Because the Metastore is in Beta, the SQL commands and Metastore formats may 
change in the next release.
-{% include startnote.html %}In Drill 1.17, this feature is supported for 
Parquet tables only and is disabled by default.{% include endnote.html %}
+{% include startnote.html %}In Drill 1.17, this feature is supported for 
Parquet tables only and is disabled by default.
+Starting from Drill 1.18, this feature is supported for all **format** plugins 
except for MaprDB.
+{% include endnote.html %}
 
 Review comment:
   Looks like the note for Drill 1.17 was removed? Should we keep it since 
Drill 1.17 is the currently-released version?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2030: Update docs for Metastore to point that all format plugins are supported

2020-03-20 Thread GitBox

paul-rogers commented on a change in pull request #2030: Update docs for 
Metastore to point that all format plugins are supported
URL: https://github.com/apache/drill/pull/2030#discussion_r395774025
 
 

 ##
 File path: _docs/sql-reference/sql-commands/021-create-schema.md
 ##
 @@ -209,13 +220,23 @@ Values are trimmed when converting to any type, except 
for varchar.
 
 ## Usage Notes 
 
-### General Information  
-- Schema provisioning only works with tables defined as directories because 
Drill must have a place to store the schema file. The directory can contain one 
or more files.  
+### General Information
+- Schema provisioning is supported only for the file system (dfs-based) 
storage plugins. It works by placing a file `.drill.schema` in the root folder 
of tables defined as a directory. The directory can contain any number of files 
(even just one) in addition to the schema file.
 
 Review comment:
   Schema provisioning works only with the file system...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2030: Update docs for Metastore to point that all format plugins are supported

2020-03-20 Thread GitBox

paul-rogers commented on a change in pull request #2030: Update docs for 
Metastore to point that all format plugins are supported
URL: https://github.com/apache/drill/pull/2030#discussion_r395775651
 
 

 ##
 File path: _docs/sql-reference/sql-commands/021-create-schema.md
 ##
 @@ -446,7 +467,8 @@ Note that date, time type conversion uses the Joda time 
library, thus the format
 
 
 ## Limitations
-This feature is currently in the alpha phase (preview, experimental) for Drill 
1.16 and only applies to text (CSV) files in this release. You must enable this 
feature through the `exec.storage.enable_v3_text_reader` and 
`store.table.use_schema_file` system/session options.
+This feature applies to format plugins that use the `Enhanced Vector 
Framework`. You must enable this feature through
 
 Review comment:
   ((Only developers know about EVF. Maybe:))
   
   Schema provisioning works with selected readers. If you develop a format 
plugin, you must use the `Enhanced Vector Framework` (rather than the "classic" 
techniques) to enable schema support.
   
   To use schema provisioning, you must first enable it with the ... option.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

31 matches

Mail list logo