Hi all,As part of our experimentations and use cases, we found that the need of
creating a view on top of each parquet file containing all the metadata already
specifyed earlier as a source limitation in the process of making data
available to Tableau. Using the parquet footer would considered to be a huge
improvement in this area.RegsBoris
Le Jeudi 1 octobre 2015 19h38, Jacques Nadeau <[email protected]> a écrit
:
Hey Chris,
Yes, this is definitely something that should be part of Drill but isn't
yet. REFRESH TABLE METADATA won't actually resolve the issue. There are
three separate pieces that will need to get resolved:
- Automatically scan workspace directories for known files/tables as part
of show tables/information schema (probably only workspace root files and
one directory down unless a ctas was done)
- Use parquet footer reading to expose table definition and validate query
and functions.
- Update Drill to merge different parquet schemas using schemaless rules
(and add appropriate projections to scans as necessary)
We should put together some jiras for each of these and get them done.
Thanks for bringing this up, it is something that should be fixed.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Sep 28, 2015 at 6:25 AM, Chris Mathews <[email protected]> wrote:
> Hi
>
> Being new to Drill I am working on a capabilities study to store telecoms
> probe data as parquet files on an HDFS server, for later analysis and
> visualisation using Tableau Desktop/Server with Drill and Zookeeper via
> ODBC/JDBC etc.
>
> We store the parquet files on the HDFS server using an in-house ETL
> platform, which amongst other things transforms the massive volumes of
> telecoms probe data into millions of parquet files, writing out the parquet
> files directly to HDFS using AvroParquetWriter. The probe data arrives at
> regular intervals (5 to 15 minutes; configurable), so for performance
> reasons we use this direct AvroParquetWriter approach rather than writing
> out intermediate files and loading them via the Drill CTAS route.
>
> There has been some success, together with some frustration. After
> extensive experimentation we have come to the conclusion that to access
> these parquet files using Tableau we have to configure Drill with
> individual views for each parquet schema, and cast the columns to specific
> data types before Tableau can access the data correctly.
>
> This is a surprise as I thought Drill would have some way of exporting the
> schemas to Tableau having defined AVRO schemas for each parquet file, and
> the parquet files storing the schema as part of the data. We now find we
> have to generate schema definitions in AVRO for the AvroParquetWriter
> phase, and also a Drill view for each schema to make them visible to
> Tableau.
>
> Also, as part of our experimentation we did create some parquet files
> using CTAS. The directory is created and the files contain the data but the
> tables do not seem to be displayed when we do a SHOW TABLES command.
>
> Are we correct in our thinking about Tableau requiring views to be
> created, or have we missed something obvious here ?
>
> Will the new REFRESH TABLE METADATA <path to table> feature (Drill 1.2 ?)
> help us when it becomes available ?
>
> Help and suggestions much appreciated.
>
> Cheers -- Chris
>
>
