Awesome, thanks! -- Jacques Nadeau CTO and Co-Founder, Dremio
On Sun, Aug 30, 2015 at 6:52 AM, Edmon Begoli <[email protected]> wrote: > Thanks, Jacques. I might add your answers in a Javadoc and also maybe write > an addendum to the documentation as I just told Ted. > > It helps me learn and it will benefit others. > > On Saturday, August 29, 2015, Jacques Nadeau <[email protected]> wrote: > > > supportsRead -> Drill can read a table from this format (e.g. read a json > > file) > > supportsWrite -> Drill can write a table to this format (e.g. write a > json > > file) > > > > question: -- what does StoragePluginOptimizerRule do? > > public Set<StoragePluginOptimizerRule> > > getOptimizerRules(OptimizerRulesContext optimizerContext); > > > > You can generally return an empty set for this. You could possibly > expose > > one or more specialized rules for this format that the Drill query > > optimizer would leverage. For example, if your plugin supported built-in > > aggregations, you could expose a rule that rewrote a aggregate < > your_scan > > to be simply a your_scan_with_aggregation. > > > > public boolean isBlockSplittable() > > Whether or not your format can be arbitrarily split and handle a random > > start point. For example, a csv file supports this since our reader can > > start in the middle of a line and stop at the next line beyond a given > > split point. ON the flipside, Parquet can't use arbitrary split points. > > JSON also doesn't support arbitrary split points since (in the most > general > > case), there is no way to search to the start of the next record. > > > > question: -- what criteria makes this truth; What does this mean? > > public boolean isCompressible() > > > > Whether this format could also be in a compression container. For > example: > > csv.gz versus csv. This is an external compression container that would > be > > handled outside of your format code. Parquet also wouldn't support this > > since it has an internal compression scheme. > > > > > > question: -- is this configuration that is specified in the "Storage" > > section on the web ui/config. files? > > public StoragePluginConfig getConfig(); > > > > Yes, should be a jackson serializable object that extends > > StoragePluginConfig. What is input when configuring in the web ui. > > > > > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Sat, Aug 29, 2015 at 7:01 PM, Edmon Begoli <[email protected] > > <javascript:;>> wrote: > > > > > On *StoragePlugin*: > > > > > > question: -- I assume this means that sources of this storage format > kind > > > can be read from the original sources, correct? > > > public boolean supportsRead(); > > > > > > question: -- Is this writing to the original source or does this mean > > > creation of the table for this format? > > > public boolean supportsWrite(); > > > > > > question: -- what does StoragePluginOptimizerRule do? > > > public Set<StoragePluginOptimizerRule> > > > getOptimizerRules(OptimizerRulesContext optimizerContext); > > > > > > question: -- is this configuration provided on the web ui or config. > > files? > > > public StoragePluginConfig getConfig(); > > > > > > > > > On *EasyFormatPlugin*: > > > > > > question: -- what criteria makes this truth; What does this mean? > > > public boolean isBlockSplittable() > > > > > > question: -- what criteria makes this truth; What does this mean? > > > public boolean isCompressible() > > > > > >
