Hi JC,
Like Paul suggested the default values must be added to drill-module.conf
file. Please follow these steps to add a new system/session option:
Step-1 : Add a validator in the ExecConstants.java similar to Kafka reader
*public static final String MSGPACK_LEARN_SCHEMA = *
*"store.msgpack.reader.learnschema"; BooleanValidator
MSGPACK_LEARN_SCHEMA_VALIDATOR = new
BooleanValidator(MSGPACK_LEARN_SCHEMA, null);*
Step-2: Add the validator from the above step to the
*definitions* in SystemOptionManager.java
Step-3 : Add the option in the conf file (drill-module.conf or
drill-override.conf file ) under the name space drill.exec.options
*drill.exec.options : {*
* store.msgpack.reader.learnschema : true*
* }*
Step-4: Set the default value for the option in the conf file as shown above
Step-5: OptionManager loads default values from the conf file, user
can override the default values by issuing alter SYSTEM/SESSION.
Thank you,
Jyothsna
On Sun, Oct 7, 2018 at 9:43 PM Paul Rogers <[email protected]>
wrote:
> Hi JC,
>
> Unless something has changed recently, it turns out that system/session
> options are global: they must be defined in the one big file you
> discovered, and default values must be listed in the master
> drill-module.conf file.
>
> It would be a handy feature to modify this to allow modules to add
> options. Easy to define the defaults using the drill-module.conf file for
> your module. But, we'd need some way for Drill to invoke code in your
> module that would populate the master options list.
>
> Jyothsna, you did a lot of work on options. Any ideas for JC on how to
> solve the problems he outlined?
>
> Thanks,
> - Paul
>
>
>
> On Sunday, October 7, 2018, 6:18:09 AM PDT, Jean-Claude Cote <
> [email protected]> wrote:
>
> Hi
>
> I'm writing a new msgpack data source for Drill. I would like to be able to
> configure the reader using the alter session mechanism using sqlline,
> something like "alter session set `store.msgpack.reader.learnschema` =
> true".
>
> However I'm unable to make this work. In my format plugin I have added this
> code to my MsgpackRecordReader. Note I'm taking inspiration from the
> JSONRecordReader.
>
> public static final String MSGPACK_LEARN_SCHEMA =
> "store.msgpack.reader.learnschema";
> public static final BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR = new
> BooleanValidator(MSGPACK_LEARN_SCHEMA, null);
>
>
> public MsgpackRecordReader(MsgpackFormatConfig config, final
> FragmentContext fragmentContext, final String inputPath,
> final DrillFileSystem fileSystem, final List<SchemaPath> columns) {
>
> Preconditions.checkArgument((inputPath != null), "InputPath must be
> set.");
> this.hadoopPath = new Path(inputPath);
>
> this.fileSystem = fileSystem;
> this.fragmentContext = fragmentContext;
> this.learnSchema =
> fragmentContext.getOptions().getOption(MSGPACK_LEARN_SCHEMA_VALIDATOR);
> setColumns(columns);
> }
>
> However this is not working. I noticed that the JSONRecordReader registers
> it's properties in the SystemOptionManager like so. Is there a way for a
> format plugin in the "contrib" folder to be able to do the same?
>
> public class SystemOptionManager extends BaseOptionManager implements
> AutoCloseable {
>
> /**
> * Creates all the OptionDefinitions to be registered with the {@link
> SystemOptionManager}.
> * @return A map
> */
> public static CaseInsensitiveMap<OptionDefinition>
> createDefaultOptionDefinitions() {
> new
> OptionDefinition(ExecConstants.JSON_SKIP_MALFORMED_RECORDS_VALIDATOR),
>
> Thank you
> jc
>
