-----Formatting the previous email--------
Hi JC,
Like Paul suggested the default values must be added to drill-module.conf
file. Please follow these steps to add a new system/session option:
Step-1 : Add a validator in the ExecConstants.java similar to Kafka reader
*public static final String MSGPACK_LEARN_SCHEMA =
"store.msgpack.reader.learnschema"; *
* BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR = new
BooleanValidator(MSGPACK_LEARN_SCHEMA, null);*
Step-2: Add the validator from the above step to the *definitions* in
SystemOptionManager.java
Step-3 : Add the option in the conf file (drill-module.conf or
drill-override.conf file ) under the name space drill.exec.options
*drill.exec.options : { store.msgpack.reader.learnschema :
true }*
Step-4: Set the default value for the option in the conf file as shown above
Step-5: OptionManager loads default values from the conf file, user can
override the default values by issuing alter SYSTEM/SESSION.
Thank you,
Jyothsna
On Mon, Oct 8, 2018 at 1:58 PM Venkata Donapati <[email protected]> wrote:
> Hi JC,
> Like Paul suggested the default values must be added to drill-module.conf
> file. Please follow these steps to add a new system/session option:
>
> Step-1 : Add a validator in the ExecConstants.java similar to Kafka reader
>
>
>
> *public static final String MSGPACK_LEARN_SCHEMA = *
> *"store.msgpack.reader.learnschema"; BooleanValidator
> MSGPACK_LEARN_SCHEMA_VALIDATOR = new
> BooleanValidator(MSGPACK_LEARN_SCHEMA, null);*
>
> Step-2: Add the validator from the above step to the
> *definitions* in SystemOptionManager.java
>
> Step-3 : Add the option in the conf file (drill-module.conf or
> drill-override.conf file ) under the name space drill.exec.options
>
>
> *drill.exec.options : {*
>
> * store.msgpack.reader.learnschema : true*
>
> * }*
>
> Step-4: Set the default value for the option in the conf file as shown
> above
>
> Step-5: OptionManager loads default values from the conf file, user
> can override the default values by issuing alter SYSTEM/SESSION.
>
>
>
> Thank you,
>
> Jyothsna
>
>
> On Sun, Oct 7, 2018 at 9:43 PM Paul Rogers <[email protected]>
> wrote:
>
> > Hi JC,
> >
> > Unless something has changed recently, it turns out that system/session
> > options are global: they must be defined in the one big file you
> > discovered, and default values must be listed in the master
> > drill-module.conf file.
> >
> > It would be a handy feature to modify this to allow modules to add
> > options. Easy to define the defaults using the drill-module.conf file for
> > your module. But, we'd need some way for Drill to invoke code in your
> > module that would populate the master options list.
> >
> > Jyothsna, you did a lot of work on options. Any ideas for JC on how to
> > solve the problems he outlined?
> >
> > Thanks,
> > - Paul
> >
> >
> >
> > On Sunday, October 7, 2018, 6:18:09 AM PDT, Jean-Claude Cote <
> > [email protected]> wrote:
> >
> > Hi
> >
> > I'm writing a new msgpack data source for Drill. I would like to be able
> to
> > configure the reader using the alter session mechanism using sqlline,
> > something like "alter session set `store.msgpack.reader.learnschema` =
> > true".
> >
> > However I'm unable to make this work. In my format plugin I have added
> this
> > code to my MsgpackRecordReader. Note I'm taking inspiration from the
> > JSONRecordReader.
> >
> > public static final String MSGPACK_LEARN_SCHEMA =
> > "store.msgpack.reader.learnschema";
> > public static final BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR =
> new
> > BooleanValidator(MSGPACK_LEARN_SCHEMA, null);
> >
> >
> > public MsgpackRecordReader(MsgpackFormatConfig config, final
> > FragmentContext fragmentContext, final String inputPath,
> > final DrillFileSystem fileSystem, final List<SchemaPath> columns) {
> >
> > Preconditions.checkArgument((inputPath != null), "InputPath must be
> > set.");
> > this.hadoopPath = new Path(inputPath);
> >
> > this.fileSystem = fileSystem;
> > this.fragmentContext = fragmentContext;
> > this.learnSchema =
> > fragmentContext.getOptions().getOption(MSGPACK_LEARN_SCHEMA_VALIDATOR);
> > setColumns(columns);
> > }
> >
> > However this is not working. I noticed that the JSONRecordReader
> registers
> > it's properties in the SystemOptionManager like so. Is there a way for a
> > format plugin in the "contrib" folder to be able to do the same?
> >
> > public class SystemOptionManager extends BaseOptionManager implements
> > AutoCloseable {
> >
> > /**
> > * Creates all the OptionDefinitions to be registered with the {@link
> > SystemOptionManager}.
> > * @return A map
> > */
> > public static CaseInsensitiveMap<OptionDefinition>
> > createDefaultOptionDefinitions() {
> > new
> > OptionDefinition(ExecConstants.JSON_SKIP_MALFORMED_RECORDS_VALIDATOR),
> >
> > Thank you
> > jc
> >
>
