drill-jdbc-all-1.7.0-SNAPSHOT.jar max size

[Arina Yelchiyeva]

Hi all!

Drill has enforcer for drill-jdbc-all-1.7.0-SNAPSHOT.jar max size. Max size
is 2000.
Currently on master jar size is 19956787.
43213 bytes is left till the limit. I have exceeded this limit just with
adding a of couple of new  classes.

I am going to create Jira to update this limit.
Just wanted to know your opinion on new max size. 3000 will be ok?


Kind regards
Arina

Dynamic UDFs support

[Arina Yelchiyeva]

Hi all!

I have created Jira to allow dynamic UDFs support in Drill (
https://issues.apache.org/jira/browse/DRILL-4726). There is a link to
design document in Jira description.
Comments or suggestions are welcomed.

Kind regards
Arina

Re: Dynamic UDFs support

[Arina Yelchiyeva]

t; If we want a very simple solution that requires minimal change,
> perhaps
> >> we
> >>>> can use an even simpler solution. In the proposed design, the user
> still
> >>>> must distribute code to all the nodes. The primary change is to tell
> >> Drill
> >>>> to load (or unload) that code. Can accomplish the same result easier
> >> simply
> >>>> by having Drill periodically scan certain directories looking for new
> >> (or
> >>>> removed) jars? Still won’t work with YARN, or solve the name space
> >> issues,
> >>>> but will work for existing non-YARN Drill users without new SQL
> syntax.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Paul
> >>>>
> >>>>> On Jun 16, 2016, at 2:07 PM, Jacques Nadeau <jacq...@dremio.com>
> >> wrote:
> >>>>>
> >>>>> Two quick thoughts:
> >>>>>
> >>>>> - (user) In the design document I didn't see any discussion of
> >>>>> ownership/conflicts or unloading. Would be helpful to see the
> thinking
> >>>> there
> >>>>> - (dev) There is a row oriented facade via the
> >>>>> FieldReader/FieldWriter/ComplexWriter classes. That would be a good
> >> place
> >>>>> to start when trying to implement an alternative interface.
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Jacques Nadeau
> >>>>> CTO and Co-Founder, Dremio
> >>>>>
> >>>>> On Thu, Jun 16, 2016 at 11:32 AM, John Omernik <j...@omernik.com>
> >> wrote:
> >>>>>
> >>>>>> Honestly, I don't see it as a priority issue. I think some of the
> >> ideas
> >>>>>> around community java UDFs could be a better approach. I'd hate to
> >> take
> >>>>>> away from other work to hack in something like this.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jun 16, 2016 at 1:19 PM, Paul Rogers <prog...@maprtech.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Ted refers to source code transformation. Drill gains its speed
> from
> >>>>>> value
> >>>>>>> vectors. However, VVs are a far cry from the row-based interface
> that
> >>>>>> most
> >>>>>>> mere mortals are accustomed to using. Since VVs are very type
> >> specific,
> >>>>>>> code is typically generated to handle the specifics of each type.
> >>>>>> Accessing
> >>>>>>> VVs in Jython may be a bit of a challenge because of the "impedence
> >>>>>>> mismatch" between how VVs work and the row-and-column view expected
> >> by
> >>>>>> most
> >>>>>>> (non-Drill) developers.
> >>>>>>>
> >>>>>>> I wonder if we've considered providing a row-oriented "facade" that
> >> can
> >>>>>> be
> >>>>>>> used by roll-your own data sources and user-defined row transforms?
> >>>> Might
> >>>>>>> be a hiccup in the fast VV pipeline, but might be handy for users
> >>>> willing
> >>>>>>> to trade a bit of speed for convenience. With such a facade, the
> >> Jython
> >>>>>> row
> >>>>>>> transforms that John mentions could be quite simple.
> >>>>>>>
> >>>>>>> On Thu, Jun 16, 2016 at 10:36 AM, Ted Dunning <
> ted.dunn...@gmail.com
> >>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Since UDF's use source code transformation, using Jython would be
> >>>>>>>> difficult.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Jun 16, 2016 at 9:42 AM, Arina Yelchiyeva <
> >>>>>>>> arina.yelchiy...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Charles,
> >>>>>>>>>
> >>>>>>>>> not that I am aware of. Proposed solution doesn't invent anything
> >>>>>> new,
> >>>>>>>> just
> >>>>>>>>> adds possibility to add UDFs without drillbit restart. But
> >>>>>>> contributions
> >>>>>>>>> are welcomed.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 16, 2016 at 4:52 PM Charles Givre <cgi...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Arina,
> >>>>>>>>>> Has there been any discussion about making it possible via
> Jython
> >>>>>> or
> >>>>>>>>>> something for users to write simple UDFs in Python?
> >>>>>>>>>> My ideal would be to have this capability integrated in the web
> >> GUI
> >>>>>>>> such
> >>>>>>>>>> that a user could write their UDF (in Python) right there,
> submit
> >>>>>> it
> >>>>>>>> and
> >>>>>>>>> it
> >>>>>>>>>> would be deployed to Drill if it passes validation tests.
> >>>>>>>>>> —C
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Jun 16, 2016, at 09:34, Arina Yelchiyeva <
> >>>>>>>>> arina.yelchiy...@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi all!
> >>>>>>>>>>>
> >>>>>>>>>>> I have created Jira to allow dynamic UDFs support in Drill (
> >>>>>>>>>>> https://issues.apache.org/jira/browse/DRILL-4726). There is a
> >>>>>> link
> >>>>>>>> to
> >>>>>>>>>>> design document in Jira description.
> >>>>>>>>>>> Comments or suggestions are welcomed.
> >>>>>>>>>>>
> >>>>>>>>>>> Kind regards
> >>>>>>>>>>> Arina
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
>

Re: Dynamic UDFs support

[Arina Yelchiyeva]

4. Authorization model mentioned by Julia and John
If user won't have rights to copy jars to UDF classpath, which can be
restricted by file system, he won't be able to do much harm by running
CREATE command. If UDFs from jar were already registered, CREATE statement
will fail. CREATE OR REPLACE will just re-register UDFs.
But DELETE command is not safe. If user knows jar name, he can delete all
associated with it UDFs, as well as the binary and source jars. That's
where we'll probably need to impose restrictions.

On Tue, Jun 21, 2016 at 7:34 PM Arina Yelchiyeva <arina.yelchiy...@gmail.com>
wrote:

> 1. DELETE command - I missed to indicate it document but had it in my
> mind. When user issues DELETE command, all UDF associated with indicated
> jar is removed from DrillFunctionRegistry. And then binary and source
> files are also deleted from UDF classpath.
>
> 2. Distribution race condition described by Paul
> User issues CREATE command and gets confirmation that UDFs is registered
> only if all drilllbits have confirmed that registration was successful.
> I don't expect user to start using UDFs in queries prior to CREATE command
> success / failure result, which is possible but strange.
>
> 3. DoY
> @Paul
> If instead of using $DRILL_HOME/jars/3rdparty/udf directly we use
> $DRILL_UDF environment variable which will be set during drillbit start
> (like $DRILL_LOG_DIR). Location stored in this variable will be added to
> Drill classpath during start.
> Will it ease DoY integration somehow?
>
> Kind regards
> Arina
>
> On Tue, Jun 21, 2016 at 7:15 PM yuliya Feldman <yufeld...@yahoo.com.invalid>
> wrote:
>
>> Just thoughts:
>> You can try to reuse distributed cache Let Drill AM do the needful in
>> terms of orchestrating UDF jars distribution.
>> But
>> I would be inclined to have a common path that is independent of the fact
>> that it is Drill on YARN or not, as maintaining two separate ways of
>> dealing with loading/unloading UDFs will be painful and error prone.
>> One more note (I left a comment in the doc) - not sure about
>> authorization model here - we need to have some.
>> Just my 2cThanks
>>
>>   From: Paul Rogers <prog...@maprtech.com>
>>  To: "dev@drill.apache.org" <dev@drill.apache.org>
>>  Sent: Monday, June 20, 2016 7:32 PM
>>  Subject: Re: Dynamic UDFs support
>>
>> Hi Neeraja,
>>
>> The proposal calls for the user to copy the jar file to each Drillbit
>> node. The jar would go into a new $DRILL_HOME/jars/3rdparty/udf directory.
>>
>> In Drill-on-YARN (DoY), YARN is responsible for copying Drill code to
>> each node (which is good.) YARN puts that code in a location known only to
>> YARN. Since the location is private to YARN, the user can’t easily hunt
>> down the location in order to add the udf jar. Even if the user did find
>> the location, the next Drillbit to start would create a new copy of the
>> Drill software, without the udf jar.
>>
>> Second, in DoY we have separated user files from Drill software. This
>> makes it much easier to distribute the software to each node: we give the
>> Drill distribution tar archive to YARN, and YARN copies it to each node and
>> untars the Drill files. We make a separate copy of the (far smaller) set of
>> user config files.
>>
>> If the udf jar goes into a Drill folder ($DRILL_HOME/jars/3rdparty/udf),
>> then the user would have to rebuild the Drill tar file each time they add a
>> udf jar. When I tried this myself when building DoY, I found it to be slow
>> and error-prone.
>>
>> So, the solution is to place the udf code in the new “site” directory:
>> $DRILL_SITE/jars. That’s what that is for. Then, let DoY automatically
>> distribute the code to every node. Perfect! Except that it does not work to
>> dynamically distribute code after Drill starts.
>>
>> For DoY, the solution requirements are:
>>
>> 1. Distribute code using Drill itself, rather than manually copying jars
>> to (unknown) Drill directories.
>> 2. Ensure the solution works even if another Drillbit is spun up later,
>> and uses the original Drill tar file.
>>
>> I’m thinking we want to leverage DFS: place udf files into a well-known
>> DFS directory. Register the udf into, say, ZK. When a new Drillbit starts,
>> it looks for new udf jars in ZK, copies the file to a temporary location,
>> and launches. An existing Drill is notified of the change and does the same
>> download process. Clean-up is needed at some point to remove ZK entries if
>> the udf jar becomes statically available on the next launch. That needs
>> more though

Re: Dynamic UDFs support

[Arina Yelchiyeva]

Hi Charles,

not that I am aware of. Proposed solution doesn't invent anything new, just
adds possibility to add UDFs without drillbit restart. But contributions
are welcomed.

On Thu, Jun 16, 2016 at 4:52 PM Charles Givre <cgi...@gmail.com> wrote:

> Arina,
> Has there been any discussion about making it possible via Jython or
> something for users to write simple UDFs in Python?
> My ideal would be to have this capability integrated in the web GUI such
> that a user could write their UDF (in Python) right there, submit it and it
> would be deployed to Drill if it passes validation tests.
> —C
>
>
> > On Jun 16, 2016, at 09:34, Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> wrote:
> >
> > Hi all!
> >
> > I have created Jira to allow dynamic UDFs support in Drill (
> > https://issues.apache.org/jira/browse/DRILL-4726). There is a link to
> > design document in Jira description.
> > Comments or suggestions are welcomed.
> >
> > Kind regards
> > Arina
>
>

Re: Dynamic UDFs support

[Arina Yelchiyeva]

Hi all!

I have updated design document.
Main changes:
1. Add to Drill’s config цшер  the staging and registration DFS locations.
2. User is no longer is responsible for copying jars into drillbit nodes.
Now user needs to copy jars into staging DFS location from where drillbits
will copy them to local fs.
2. During UDFs registration jars will be moved to DFS registration area.
3. During start up drillbit will copy all jars from registration area, so
newly added drillbit will have all UDFs as others.
4. Security issues - probably they will be added later as enhancement.

More detains in the document:
https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit

Kind regards
Arina

On Fri, Jun 17, 2016 at 1:25 AM Paul Rogers <prog...@maprtech.com> wrote:

> Hi All,
>
> To answer Arina on item 3: there is actually no good location on any local
> node to put the UDFs. Reason: DoY allows the admin to start a Drillbit on
> any available node. When it starts, a new, fresh copy of Drill will be
> downloaded, and this can happen after the user issued the CREATE command.
>
> What we need is a shared, secure distributed storage location from which
> Drillbits can download the needed jar files. Something like… DFS! Indeed,
> this is how YARN stores the Drill archive from which it creates the Drill
> install directory on each node. We can’t quite use YARN’s mechanism (YARN
> is aware only of the files uploaded when launching an app), but we can do
> something similar.
>
> So, brainstorming a bit…
>
> 1. Store the UDF jar in a pre-defined DFS location.
>
> 2. The CREATE function 1) uploads the jar to the DFS location, and 2)
> creates some kind of registry entry.
>
> 3. The DELETE function 1) deregisters the jar (and function), but 2) does
> not delete the jar (this allows in-flight queries to complete.)
>
> 3. Drillbits periodically check DFS for changed registrations, downloading
> any needed jars. (YARN, Spark, Storm and others already do something
> similar.)
>
> 4. Registry check is “forced” when processing a query with a function that
> is not currently registered. (Doing so resolves any possible race
> conditions.)
>
> 5. Some process (perhaps time based) removes old, unregistered jar files.
> (Or, we could get fancy and use reference counts. The reference count would
> be required if the user wants to delete, then recreate, the same function
> and jar to avoid conflict with in-flight queries.)
>
> We can build security on this as follows:
>
> 1. Define permissions for who can write to the DFS location. Or, indeed,
> have subdirectories by user and grant each user permission only on their
> own UDF directory.
>
> 2. Provide separate registries for per-user functions (private) and global
> functions (public). Only the admin can add global functions. But, only the
> user that uploads a private function can use it.
>
> 3. Leverage the Java class loader to isolate UDFs in their own name space
> (see Eclipse & Tomcat for examples). That is, Drill can call into a UDF,
> UDFs can call selected Drill code, but UDFs can’t shadow Drill classes
> (accidentally or maliciously.) Plus, my function Foo won’t clash with your
> function Foo if both are private.
>
> Sorry that this has wandered a bit far from the original simple design,
> but the above may capture much of what folks expect in modern distributed
> big data systems.
>
> I wonder if a good next step might be to review the notes in the design
> doc, in the JIRA, and in this e-mail chain and to prepare a summary of
> technical requirements, and a proposed design. Postpone, at least for now,
> concerns about the amount of work; we can worry about that once folks agree
> on your revised design.
>
> Thanks,
>
> - Paul
>
>
> > On Jun 21, 2016, at 9:48 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
> >
> > 4. Authorization model mentioned by Julia and John
> > If user won't have rights to copy jars to UDF classpath, which can be
> > restricted by file system, he won't be able to do much harm by running
> > CREATE command. If UDFs from jar were already registered, CREATE
> statement
> > will fail. CREATE OR REPLACE will just re-register UDFs.
> > But DELETE command is not safe. If user knows jar name, he can delete all
> > associated with it UDFs, as well as the binary and source jars. That's
> > where we'll probably need to impose restrictions.
> >
> > On Tue, Jun 21, 2016 at 7:34 PM Arina Yelchiyeva <
> arina.yelchiy...@gmail.com>
> > wrote:
> >
> >> 1. DELETE command - I missed to indicate it document but had it in my
> >> mind. When user issues DELETE command, all UDF associated with indicated
> &

Re: TIMESTAMPADD sql jdbc function execution in drill

[Arina Yelchiyeva]

Hi Sudip,

timestampadd / timestampdiff functions are not currently implemented in
Drill but they are in development.
See Jira https://issues.apache.org/jira/browse/DRILL-3610


Kind regards
Arina

On Mon, Feb 8, 2016 at 9:29 AM Sudip Mukherjee 
wrote:

> Thanks Khurram.
>
> I am not sure if I am looking at the correct place. Seeing it in calcite,
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/SqlJdbcFunctionCall.java
>
> I'll try to check further on the date-time functions.
>
> Thanks,
> Sudip
>
> -Original Message-
> From: Khurram Faraaz [mailto:kfar...@maprtech.com]
> Sent: 08 February 2016 PM 12:14
> To: dev@drill.apache.org
> Subject: Re: TIMESTAMPADD sql jdbc function execution in drill
>
> I didn't find it on the documented date time functions here
> https://drill.apache.org/docs/date-time-functions-and-arithmetic/
>
> On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee  >
> wrote:
>
> > Hi,
> > Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I
> > am getting an error while running the below query
> >
> > SELECT CAST(EXTRACT(YEAR FROM CAST({fn
> > TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> > '1970-01-01'})} AS DATE)) AS INTEGER) AS
> > `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> > FROM `dfs.data`.`MyFileShare_2214` `myshare` GROUP BY
> > CAST(EXTRACT(YEAR FROM CAST({fn
> > TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> > '1970-01-01'})} AS DATE)) AS INTEGER)
> >
> >
> > VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column
> > 'SQL_TSI_YEAR' not found in any table
> >
> > Thanks,
> > Sudip
> >
> >
> >
> > ***Legal Disclaimer***
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > **
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **

project build fails -> drill-jdbc-all-1.5.0-SNAPSHOT.jar is outside the expected size range

[Arina Yelchiyeva]

Hi all!

Just pulled recent changes from master (revision number
1b96174b1e5bafb13a873dd79f03467802d7c929) and mvn clean install -DskipTests
failed with the following error:

*[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce
(enforce-jdbc-jar-compactness) on project drill-jdbc-all: Some Enforcer
rules have failed. Look above for specific messages explaining why the rule
failed.*

*[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireFilesSize
failed with message:*
*The file drill-jdbc-all-1.5.0-SNAPSHOT.jar is outside the expected size
range. *

*This is likely due to you adding new dependencies to a java-exec and not
updating the excludes in this module. This is important as it minimizes the
size of the dependency of Drill application users.*
*F:\git_repo\drill\exec\jdbc-all\target\drill-jdbc-all-1.5.0-SNAPSHOT.jar
size (44664290) too large. Max. is 2000
F:\git_repo\drill\exec\jdbc-all\target\drill-jdbc-all-1.5.0-SNAPSHOT.jar*

Had to change 2000 -> 5000 in
jdbc-all pom.xml to build the project.

Do we need to create jira for this or it's already being fixed?

Kind regards
Arina

hive translate function is not working from Drill

[Arina Yelchiyeva]

Hi all!

Does all Hive functions work in Drill?

I have faced the issue with translate function.
In Hive "select translate(name, 'A', 'B') from users" works fine.
But in Drill "select translate(name, 'A', 'B') from hive.`users`" return
the following error:

org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
Encountered "," at line 1, column 22. Was expecting one of: "USING" ...
"NOT" ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ...
"<" ... "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ...
"AND" ... "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ...
"[" ... "." ... "(" ... while parsing SQL query: select translate(name,
'A', 'B') from hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f
on localhost:31010]

Am I missing something? Or I should create jira to fix for bug fix?

Kind regards
Arina

Re: hive translate function is not working from Drill

[Arina Yelchiyeva]

Though translate(expression USING identifier) is an SQL standard, translate
(expession, from_string, to_string) is widely used among other databases. I
guess it would be nice if we could allow alternative TRANSLATE syntax in
Drill.

Asked Calcite community last week if there is a way to complement Calcite
parse instructions for TRANSLATE function without Calcite code change. But
no reply so far.

Is it ok if I create Jira to keep track on this issue so at least it won't
be lost?

On Wed, Feb 24, 2016 at 11:52 PM Jinfeng Ni <jinfengn...@gmail.com> wrote:

> Checked standard SQL reference,  ISO/IEC 9075-2:2011(E), section 6.30
>
>  ::=
>   TRANSLATE  
> USING  
>
> Looks like Calcite follows the standard SQL reference.
>
>
> On Wed, Feb 24, 2016 at 1:46 PM, Jinfeng Ni <jinfengn...@gmail.com> wrote:
> > Looks like Calcite, the SQL parser that Drill uses, treats translate
> > as a build-in function : translate( expression USING identifier).
> > That's why you saw the Parser error.
> >
> >
> > [1]
> https://github.com/apache/calcite/blob/master/core/src/main/codegen/templates/Parser.jj#L3987-L4005
> >
> > On Wed, Feb 24, 2016 at 1:23 PM, Arina Yelchiyeva
> > <arina.yelchiy...@gmail.com> wrote:
> >> Hi all!
> >>
> >> Does all Hive functions work in Drill?
> >>
> >> I have faced the issue with translate function.
> >> In Hive "select translate(name, 'A', 'B') from users" works fine.
> >> But in Drill "select translate(name, 'A', 'B') from hive.`users`" return
> >> the following error:
> >>
> >> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> >> Encountered "," at line 1, column 22. Was expecting one of: "USING" ...
> >> "NOT" ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">"
> ...
> >> "<" ... "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||"
> ...
> >> "AND" ... "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET"
> ...
> >> "[" ... "." ... "(" ... while parsing SQL query: select translate(name,
> >> 'A', 'B') from hive.users ^ [Error Id:
> ba21956b-3285-4544-b3b2-fab68b95be1f
> >> on localhost:31010]
> >>
> >> Am I missing something? Or I should create jira to fix for bug fix?
> >>
> >> Kind regards
> >> Arina
>

Re: hive translate function is not working from Drill

[Arina Yelchiyeva]

Sorry, Julian, I guess I wasn't subscribed to Calcite mailing list, that's
why I have missed your reply.
I'll create appropriate Jira and see what I can do.

On Mon, Feb 29, 2016 at 8:48 PM Julian Hyde <jh...@apache.org> wrote:

> Arina: I did reply to your message on dev@calcite. See
> http://mail-archives.apache.org/mod_mbox/calcite-dev/201602.mbox/%3CDB1F2B6D-C23A-45E7-B400-C7458DCD9CF1%40apache.org%3E
> <
> http://mail-archives.apache.org/mod_mbox/calcite-dev/201602.mbox/%3cdb1f2b6d-c23a-45e7-b400-c7458dcd9...@apache.org%3E
> >.
>
> It would be great if this feature could be contributed to Calcite, not
> just Drill.
>
> Julian
>
> > On Feb 29, 2016, at 2:26 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
> >
> > Though translate(expression USING identifier) is an SQL standard,
> translate
> > (expession, from_string, to_string) is widely used among other
> databases. I
> > guess it would be nice if we could allow alternative TRANSLATE syntax in
> > Drill.
> >
> > Asked Calcite community last week if there is a way to complement Calcite
> > parse instructions for TRANSLATE function without Calcite code change.
> But
> > no reply so far.
> >
> > Is it ok if I create Jira to keep track on this issue so at least it
> won't
> > be lost?
> >
> > On Wed, Feb 24, 2016 at 11:52 PM Jinfeng Ni <jinfengn...@gmail.com>
> wrote:
> >
> >> Checked standard SQL reference,  ISO/IEC 9075-2:2011(E), section 6.30
> >>
> >>  ::=
> >>  TRANSLATE  
> >> USING  
> >>
> >> Looks like Calcite follows the standard SQL reference.
> >>
> >>
> >> On Wed, Feb 24, 2016 at 1:46 PM, Jinfeng Ni <jinfengn...@gmail.com>
> wrote:
> >>> Looks like Calcite, the SQL parser that Drill uses, treats translate
> >>> as a build-in function : translate( expression USING identifier).
> >>> That's why you saw the Parser error.
> >>>
> >>>
> >>> [1]
> >>
> https://github.com/apache/calcite/blob/master/core/src/main/codegen/templates/Parser.jj#L3987-L4005
> >>>
> >>> On Wed, Feb 24, 2016 at 1:23 PM, Arina Yelchiyeva
> >>> <arina.yelchiy...@gmail.com> wrote:
> >>>> Hi all!
> >>>>
> >>>> Does all Hive functions work in Drill?
> >>>>
> >>>> I have faced the issue with translate function.
> >>>> In Hive "select translate(name, 'A', 'B') from users" works fine.
> >>>> But in Drill "select translate(name, 'A', 'B') from hive.`users`"
> return
> >>>> the following error:
> >>>>
> >>>> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> >>>> Encountered "," at line 1, column 22. Was expecting one of: "USING"
> ...
> >>>> "NOT" ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">"
> >> ...
> >>>> "<" ... "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ...
> "||"
> >> ...
> >>>> "AND" ... "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET"
> >> ...
> >>>> "[" ... "." ... "(" ... while parsing SQL query: select
> translate(name,
> >>>> 'A', 'B') from hive.users ^ [Error Id:
> >> ba21956b-3285-4544-b3b2-fab68b95be1f
> >>>> on localhost:31010]
> >>>>
> >>>> Am I missing something? Or I should create jira to fix for bug fix?
> >>>>
> >>>> Kind regards
> >>>> Arina
> >>
>
>

drill logs in hadoop fs

[Arina Yelchiyeva]

Hi all!

Can we store drill logs in hadoop fs?
I know we can configure spill directory to be in hadoop fs. What about
drill logs (sqlline.log etc)?

Kind regards
Arina

Re: Dynamic UDFs support

[Arina Yelchiyeva]

Taking into account all previous comments and discussion we had with Parth
and Paul, please find below my design notes (I am going to prepare proper
design document, just want to see if all agree with raw version).
I propose will use lazy-init to dynamically loaded UDFs, in such case when
user issues CREATE UDF command, foreman will only validate jar and update
ZK function registry, and only if function is needed it will be loaded to
appropriate drillbit (during planning stage or fragment execution). We
might add listeners (as Paul proposed) to pre-load UDFs but I didn't
include it to current release to simplify solution but we might re-consider
this.
I have looked at issue with class loading and unloading and if we ship each
jar with its own classloader, DELETE functionality can be introduced in
current release, at least marked as experimental or for developers use
only, to ease UDF development process.

Any comments are welcomed.

*Invariants*

1. DFS staging area where user copies jar to be loaded

2. DFS udf area (former registration area) where all validated jars are
present

3. ZK function registry - contains list of all dynamically loaded UDFs and
their jars. UDF name will be represented as combination of name and input
parameters.

4. Lazy-init - all dynamically loaded UDFs will be loaded to drillbit upon
request, i.e. if drillbits receives query or fragment that contains such UDF

5. Currently only CREATE and DELETE statements are supported


*Adding UDFs*

1. User copies source and binary (hereinafter jar) to DFS staging area
2. User issues CREATE UDF command
3. Foreman receives request to create UDF:
a) checks if jar is present in staging area
b) copies jar to temporary DFS location
c) validates UDFs present in jar locally:
1) copies jar to temporary local fs
2) scans jar using temporary classloader
3) checks if there are any duplicates in local function registry
4) returns list of UDFs to be registered
d) validates UDFs present in jar in ZK:
1) takes list of dynamically loaded UDFs from ZK
2) checks if there are no duplicates either by jar name or among UDFs
3) moves jar from DFS temporary area to DFS udf area
4) updates ZK with list of new dynamic UDFs
5) removes jar from staging area
6) returns confirmation to user that UDFs were registered


*Lazy-init*

1. User issues query with dynamically loaded UDF.

2. During planning stage or fragment execution, if UDF is not present in
local function registry,  drillbit:

a) checks if such UDF is present in ZK function registry

b) if present, loads UDF using jar name, otherwise return an error

c) proceeds planning stage or fragment execution


*New drillbit registration / Drillbit re-start*

Local udf directory is re-created, to clean up previously loaded jars if any


*Delete UDF*

Each jar that going to be loaded dynamically will have its own classloader
which will solve problem with loading and unloading classes with the same
name.


1. User issues DELETE command (delete will operate on jar name level)

2. Foreman receives DELETE request:

a) checks if such jar is present in ZK function registry

b) creates ephemeral znode /udf/delete/jar_name

c) removes record in ZK function registry

d) removes jar from DFS udf area

e) removes ephemeral znode from /udf/delete/jar_name

f) returns confirmation to user that UDFs were deleted

3. Drillbits are subscribed to /udf/delete znode, when new znode with jar
name appears, drillbit:

a) removes all UDFs associated with jar name from local function registry

b) removes jar from local udf directory


*Limitations*

1. When user runs DELETE command, some queries that are using deleted UDFs
may fail during fragment execution if by that time UDF has been deleted
from local registry. Ideally, before submitting DELETE command, user needs
to make sure, no one is running queries using UDFs from that particular jar.


2. We encourage users not to delete any jars from DFS udf area manually, as
it may lead to inconsistency between ZK function registry and DFS udf area.


3. CREATE statement is not atomic in part when we copy validated jar to DFS
udf area and updating ZK function registry with list of new UDFs. In case
of failure between these two steps, some unused jars may be left in DFS udf
area but they won’t harm current process. LIST JARS command can be
introduced to show used jars.


Kind regards
Arina

On Fri, Jul 22, 2016 at 7:15 PM Keys Botzum  wrote:

> No disagreement on deferral but I raised my initial concern precisely
> because I'm concerned about the practicality of the "restart the cluster"
> option. I  sighted my concerns about laptops and development clusters.  I
> was wondering if there might be some small things Drill could do to help.
> If there is nothing that can be done to make this easier, so be it, but I
> think that's going to be a big impedance.
>
> Keys
> ___
> Keys Botzum
> Senior Principal Technologist
> kbot...@maprtech.com 

Re: Dynamic UDFs support

[Arina Yelchiyeva]

Hi all!

I have prepared updated design document (which describes approach where
DFS, Zookeeper, lazy-init, create and drop notions are used).
Please use new link -
https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing

Previous document link can be found in References section.

Kind regards
Arina

On Tue, Jul 26, 2016 at 8:35 PM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> Thank you Arina
> Yuliya
>
>   From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
>  To: dev@drill.apache.org; yuliya Feldman <yufeld...@yahoo.com>
>  Sent: Tuesday, July 26, 2016 10:11 AM
>  Subject: Re: Dynamic UDFs support
>
> Sure, I'll add this option. I'll send a link to final document once it's
> done.
>
> On Tue, Jul 26, 2016 at 8:06 PM Keys Botzum <kbot...@maprtech.com> wrote:
>
> > +1
> >
> > Keys
> > ___
> > Keys Botzum
> > Senior Principal Technologist
> > kbot...@maprtech.com <mailto:kbot...@maprtech.com>
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com <http://www.mapr.com/>
> > > On Jul 26, 2016, at 1:05 PM, yuliya Feldman
> <yufeld...@yahoo.com.INVALID>
> > wrote:
> > >
> > > I want to make sure (also will make a note in the design doc) that we
> > have an option to disable dynamic loading/unloading of UDFs until we will
> > be able to have an ability to do proper authentication AND authorization
> of
> > the user(s).
> > >
> > >  From: Arina Yelchiyeva <arina.yelchiy...@gmail.com  > arina.yelchiy...@gmail.com>>
> > > To: dev@drill.apache.org <mailto:dev@drill.apache.org>
> > > Sent: Monday, July 25, 2016 9:09 AM
> > > Subject: Re: Dynamic UDFs support
> > >
> > > My fault, agree, DROP is more appropriate.
> > > Thanks Julian!
> > >
> > > On Mon, Jul 25, 2016 at 7:07 PM Julian Hyde <jhyde.apa...@gmail.com
> > <mailto:jhyde.apa...@gmail.com>> wrote:
> > >
> > >> But don't call it DELETE. In SQL the opposite of CREATE is DROP.
> > >>
> > >> Julian
> > >>
> > >>> On Jul 25, 2016, at 8:48 AM, Keys Botzum <kbot...@maprtech.com
> > <mailto:kbot...@maprtech.com>> wrote:
> > >>>
> > >>> I like the approach to handling DELETE. This is very useful. I think
> an
> > >> implementation that does not guarantee consistent behavior is
> perfectly
> > >> fine for use that is targeted at developers that are working on UDFs.
> As
> > >> long as the docs make the intent clear this makes me very happy.
> > >>>
> > >>> I'll defer to others more expert than I on the remainder of the
> design.
> > >>>
> > >>> Keys
> > >>> ___
> > >>> Keys Botzum
> > >>> Senior Principal Technologist
> > >>> kbot...@maprtech.com <mailto:kbot...@maprtech.com>  > kbot...@maprtech.com <mailto:kbot...@maprtech.com>>
> > >>> 443-718-0098
> > >>> MapR Technologies
> > >>> http://www.mapr.com <http://www.mapr.com/> <http://www.mapr.com/ <
> > http://www.mapr.com/>>
> > >>>> On Jul 25, 2016, at 9:55 AM, Arina Yelchiyeva <
> > >> arina.yelchiy...@gmail.com <mailto:arina.yelchiy...@gmail.com>>
> wrote:
> > >>>>
> > >>>> Taking into account all previous comments and discussion we had with
> > >> Parth
> > >>>> and Paul, please find below my design notes (I am going to prepare
> > >> proper
> > >>>> design document, just want to see if all agree with raw version).
> > >>>> I propose will use lazy-init to dynamically loaded UDFs, in such
> case
> > >> when
> > >>>> user issues CREATE UDF command, foreman will only validate jar and
> > >> update
> > >>>> ZK function registry, and only if function is needed it will be
> loaded
> > >> to
> > >>>> appropriate drillbit (during planning stage or fragment execution).
> We
> > >>>> might add listeners (as Paul proposed) to pre-load UDFs but I didn't
> > >>>> include it to current release to simplify solution but we might
> > >> re-consider
> > >>>> this.
> > >>>> I have looked at issue with class loading and unloading and if we
> ship
> > >> each
> > >

Re: Dynamic UDFs support

[Arina Yelchiyeva]

My fault, agree, DROP is more appropriate.
Thanks Julian!

On Mon, Jul 25, 2016 at 7:07 PM Julian Hyde <jhyde.apa...@gmail.com> wrote:

> But don't call it DELETE. In SQL the opposite of CREATE is DROP.
>
> Julian
>
> > On Jul 25, 2016, at 8:48 AM, Keys Botzum <kbot...@maprtech.com> wrote:
> >
> > I like the approach to handling DELETE. This is very useful. I think an
> implementation that does not guarantee consistent behavior is perfectly
> fine for use that is targeted at developers that are working on UDFs. As
> long as the docs make the intent clear this makes me very happy.
> >
> > I'll defer to others more expert than I on the remainder of the design.
> >
> > Keys
> > ___
> > Keys Botzum
> > Senior Principal Technologist
> > kbot...@maprtech.com <mailto:kbot...@maprtech.com>
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com <http://www.mapr.com/>
> >> On Jul 25, 2016, at 9:55 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
> >>
> >> Taking into account all previous comments and discussion we had with
> Parth
> >> and Paul, please find below my design notes (I am going to prepare
> proper
> >> design document, just want to see if all agree with raw version).
> >> I propose will use lazy-init to dynamically loaded UDFs, in such case
> when
> >> user issues CREATE UDF command, foreman will only validate jar and
> update
> >> ZK function registry, and only if function is needed it will be loaded
> to
> >> appropriate drillbit (during planning stage or fragment execution). We
> >> might add listeners (as Paul proposed) to pre-load UDFs but I didn't
> >> include it to current release to simplify solution but we might
> re-consider
> >> this.
> >> I have looked at issue with class loading and unloading and if we ship
> each
> >> jar with its own classloader, DELETE functionality can be introduced in
> >> current release, at least marked as experimental or for developers use
> >> only, to ease UDF development process.
> >>
> >> Any comments are welcomed.
> >>
> >> *Invariants*
> >>
> >> 1. DFS staging area where user copies jar to be loaded
> >>
> >> 2. DFS udf area (former registration area) where all validated jars are
> >> present
> >>
> >> 3. ZK function registry - contains list of all dynamically loaded UDFs
> and
> >> their jars. UDF name will be represented as combination of name and
> input
> >> parameters.
> >>
> >> 4. Lazy-init - all dynamically loaded UDFs will be loaded to drillbit
> upon
> >> request, i.e. if drillbits receives query or fragment that contains
> such UDF
> >>
> >> 5. Currently only CREATE and DELETE statements are supported
> >>
> >>
> >> *Adding UDFs*
> >>
> >> 1. User copies source and binary (hereinafter jar) to DFS staging area
> >> 2. User issues CREATE UDF command
> >> 3. Foreman receives request to create UDF:
> >> a) checks if jar is present in staging area
> >> b) copies jar to temporary DFS location
> >> c) validates UDFs present in jar locally:
> >> 1) copies jar to temporary local fs
> >> 2) scans jar using temporary classloader
> >> 3) checks if there are any duplicates in local function registry
> >> 4) returns list of UDFs to be registered
> >> d) validates UDFs present in jar in ZK:
> >> 1) takes list of dynamically loaded UDFs from ZK
> >> 2) checks if there are no duplicates either by jar name or among UDFs
> >> 3) moves jar from DFS temporary area to DFS udf area
> >> 4) updates ZK with list of new dynamic UDFs
> >> 5) removes jar from staging area
> >> 6) returns confirmation to user that UDFs were registered
> >>
> >>
> >> *Lazy-init*
> >>
> >> 1. User issues query with dynamically loaded UDF.
> >>
> >> 2. During planning stage or fragment execution, if UDF is not present in
> >> local function registry,  drillbit:
> >>
> >> a) checks if such UDF is present in ZK function registry
> >>
> >> b) if present, loads UDF using jar name, otherwise return an error
> >>
> >> c) proceeds planning stage or fragment execution
> >>
> >>
> >> *New drillbit registration / Drillbit re-start*
> >>
> >> Local udf directory is re-created, to clean up previously loaded jars
> if any
> >>
&

Re: Dynamic UDFs support

[Arina Yelchiyeva]

Sure, I'll add this option. I'll send a link to final document once it's
done.

On Tue, Jul 26, 2016 at 8:06 PM Keys Botzum <kbot...@maprtech.com> wrote:

> +1
>
> Keys
> ___
> Keys Botzum
> Senior Principal Technologist
> kbot...@maprtech.com <mailto:kbot...@maprtech.com>
> 443-718-0098
> MapR Technologies
> http://www.mapr.com <http://www.mapr.com/>
> > On Jul 26, 2016, at 1:05 PM, yuliya Feldman <yufeld...@yahoo.com.INVALID>
> wrote:
> >
> > I want to make sure (also will make a note in the design doc) that we
> have an option to disable dynamic loading/unloading of UDFs until we will
> be able to have an ability to do proper authentication AND authorization of
> the user(s).
> >
> >  From: Arina Yelchiyeva <arina.yelchiy...@gmail.com  arina.yelchiy...@gmail.com>>
> > To: dev@drill.apache.org <mailto:dev@drill.apache.org>
> > Sent: Monday, July 25, 2016 9:09 AM
> > Subject: Re: Dynamic UDFs support
> >
> > My fault, agree, DROP is more appropriate.
> > Thanks Julian!
> >
> > On Mon, Jul 25, 2016 at 7:07 PM Julian Hyde <jhyde.apa...@gmail.com
> <mailto:jhyde.apa...@gmail.com>> wrote:
> >
> >> But don't call it DELETE. In SQL the opposite of CREATE is DROP.
> >>
> >> Julian
> >>
> >>> On Jul 25, 2016, at 8:48 AM, Keys Botzum <kbot...@maprtech.com
> <mailto:kbot...@maprtech.com>> wrote:
> >>>
> >>> I like the approach to handling DELETE. This is very useful. I think an
> >> implementation that does not guarantee consistent behavior is perfectly
> >> fine for use that is targeted at developers that are working on UDFs. As
> >> long as the docs make the intent clear this makes me very happy.
> >>>
> >>> I'll defer to others more expert than I on the remainder of the design.
> >>>
> >>> Keys
> >>> _______
> >>> Keys Botzum
> >>> Senior Principal Technologist
> >>> kbot...@maprtech.com <mailto:kbot...@maprtech.com>  kbot...@maprtech.com <mailto:kbot...@maprtech.com>>
> >>> 443-718-0098
> >>> MapR Technologies
> >>> http://www.mapr.com <http://www.mapr.com/> <http://www.mapr.com/ <
> http://www.mapr.com/>>
> >>>> On Jul 25, 2016, at 9:55 AM, Arina Yelchiyeva <
> >> arina.yelchiy...@gmail.com <mailto:arina.yelchiy...@gmail.com>> wrote:
> >>>>
> >>>> Taking into account all previous comments and discussion we had with
> >> Parth
> >>>> and Paul, please find below my design notes (I am going to prepare
> >> proper
> >>>> design document, just want to see if all agree with raw version).
> >>>> I propose will use lazy-init to dynamically loaded UDFs, in such case
> >> when
> >>>> user issues CREATE UDF command, foreman will only validate jar and
> >> update
> >>>> ZK function registry, and only if function is needed it will be loaded
> >> to
> >>>> appropriate drillbit (during planning stage or fragment execution). We
> >>>> might add listeners (as Paul proposed) to pre-load UDFs but I didn't
> >>>> include it to current release to simplify solution but we might
> >> re-consider
> >>>> this.
> >>>> I have looked at issue with class loading and unloading and if we ship
> >> each
> >>>> jar with its own classloader, DELETE functionality can be introduced
> in
> >>>> current release, at least marked as experimental or for developers use
> >>>> only, to ease UDF development process.
> >>>>
> >>>> Any comments are welcomed.
> >>>>
> >>>> *Invariants*
> >>>>
> >>>> 1. DFS staging area where user copies jar to be loaded
> >>>>
> >>>> 2. DFS udf area (former registration area) where all validated jars
> are
> >>>> present
> >>>>
> >>>> 3. ZK function registry - contains list of all dynamically loaded UDFs
> >> and
> >>>> their jars. UDF name will be represented as combination of name and
> >> input
> >>>> parameters.
> >>>>
> >>>> 4. Lazy-init - all dynamically loaded UDFs will be loaded to drillbit
> >> upon
> >>>> request, i.e. if drillbits receives query or fragment that contains
> >> such UDF
> >>>>

Re: [VOTE] Release Apache Drill 1.8.0 - rc0

[Arina Yelchiyeva]

+1 (non-binding)
On Windows built from source, ran in embedded mode.
Installed on cluster.
Checked some new features and run some basic queries.

On Fri, Aug 12, 2016 at 2:48 AM, Padma Penumarthy 
wrote:

> +1
>
> Installed on the cluster and ran some basic queries
> Tried few queries in embedded mode on the mac
>
> Thanks,
> Padma
>
>
> > On Aug 10, 2016, at 10:58 PM, Aman Sinha  wrote:
> >
> > +1  (binding)
> > Downloaded source as well as binary tarballs.
> > Checked git.properties and README.
> > On Linux: Built from source and ran unit tests.
> > On Mac: Installed binary release and ran in embedded mode.
> > Ran a few queries that exercise new features and looked at query profiles
> > in Web UI.
> >
> > LGTM.
> >
> > On Wed, Aug 10, 2016 at 5:44 PM, Parth Chandra 
> wrote:
> >
> >> +1
> >>
> >> Verified source tarball and checked the signatures.
> >> Built from source. (Linux)
> >> Built C++ client from source
> >> Ran some test queries
> >>
> >> Looks good.
> >>
> >>
> >>
> >> On Wed, Aug 10, 2016 at 1:19 PM, Dechang Gu  wrote:
> >>
> >>> +1.
> >>> run tpch regression test on it, LGTM.
> >>>
> >>>
> >>> On Tue, Aug 9, 2016 at 9:26 PM, Jinfeng Ni 
> >> wrote:
> >>>
>  Hello all,
> 
>  I'd like to propose the zeroth release candidate (rc0) of Apache
>  Drill, version 1.8.0. It covers a total of 37 resolved JIRAs [1].
>  Thanks to everyone who contributed to this release.
> 
>  The tarball artifacts are hosted at [2] and the maven artifacts are
> >>> hosted
>  at [3].
> 
>  This release candidate is based on commit
>  e148d66e1945bff5397f502dc75b65033bba4811 located at [4].
> 
>  The vote will be open for the next ~72 hours ending at 10:00 PM
>  Pacific, August 12, 2016.
> 
>  [ ] +1
>  [ ] +0
>  [ ] -1
> 
>  Here's my vote: +1
> 
>  Thanks,
> 
>  Jinfeng
> 
>  [1]
>  https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>  version=12334768==12313820=
>  Create_token=A5KQ-2QAV-T4JA-FDED%7Cb9b453c1c790b30e573bfa12f498
>  1c49667bee0a%7Clout
>  [2] http://home.apache.org/~jni/drill/1.8.0/rc0/
>  [3] https://repository.apache.org/content/repositories/
>  orgapachedrill-1033/
>  [4] https://github.com/jinfengni/incubator-drill/tree/1.8.0
> 
> >>>
> >>
>
>

Re: Dynamic UDFs support

[Arina Yelchiyeva]

I also agree on using Zookeeper. I have re-worked dynamic UDF support
document taking into account Zookeeper usage.

Link to the document -
https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit

Kind regards
Arina

On Tue, Jun 28, 2016 at 12:55 AM Paul Rogers <prog...@maprtech.com> wrote:

> Great idea! We already use ZK to track storage plugins. ZK is perhaps
> better suited to register each jar and/or function that using files in DFS.
> Still need to work out the proper sequencing. But you are right, this is
> the kind of thing that ZK is supposed to solve.
>
> - Paul
>
>
> > On Jun 27, 2016, at 2:01 PM, Parth Chandra <par...@apache.org> wrote:
> >
> > Reading thru some of Paul's comments on maintaining a consistent state
> for
> > the registration of the UDF, it looks like we need a consensus protocol
> for
> > determining that all the Drillbits have the UDF deployed.
> > I believe Zookeeper can provide a stronger guarantee than a 2 phase
> > approach. Should we look into that?
> >
> > On Fri, Jun 24, 2016 at 10:00 AM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> >> Hi all!
> >>
> >> I have updated design document.
> >> Main changes:
> >> 1. Add to Drill’s config цшер  the staging and registration DFS
> locations.
> >> 2. User is no longer is responsible for copying jars into drillbit
> nodes.
> >> Now user needs to copy jars into staging DFS location from where
> drillbits
> >> will copy them to local fs.
> >> 2. During UDFs registration jars will be moved to DFS registration area.
> >> 3. During start up drillbit will copy all jars from registration area,
> so
> >> newly added drillbit will have all UDFs as others.
> >> 4. Security issues - probably they will be added later as enhancement.
> >>
> >> More detains in the document:
> >>
> >>
> https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit
> >>
> >> Kind regards
> >> Arina
> >>
> >> On Fri, Jun 17, 2016 at 1:25 AM Paul Rogers <prog...@maprtech.com>
> wrote:
> >>
> >>> Hi All,
> >>>
> >>> To answer Arina on item 3: there is actually no good location on any
> >> local
> >>> node to put the UDFs. Reason: DoY allows the admin to start a Drillbit
> on
> >>> any available node. When it starts, a new, fresh copy of Drill will be
> >>> downloaded, and this can happen after the user issued the CREATE
> command.
> >>>
> >>> What we need is a shared, secure distributed storage location from
> which
> >>> Drillbits can download the needed jar files. Something like… DFS!
> Indeed,
> >>> this is how YARN stores the Drill archive from which it creates the
> Drill
> >>> install directory on each node. We can’t quite use YARN’s mechanism
> (YARN
> >>> is aware only of the files uploaded when launching an app), but we can
> do
> >>> something similar.
> >>>
> >>> So, brainstorming a bit…
> >>>
> >>> 1. Store the UDF jar in a pre-defined DFS location.
> >>>
> >>> 2. The CREATE function 1) uploads the jar to the DFS location, and 2)
> >>> creates some kind of registry entry.
> >>>
> >>> 3. The DELETE function 1) deregisters the jar (and function), but 2)
> does
> >>> not delete the jar (this allows in-flight queries to complete.)
> >>>
> >>> 3. Drillbits periodically check DFS for changed registrations,
> >> downloading
> >>> any needed jars. (YARN, Spark, Storm and others already do something
> >>> similar.)
> >>>
> >>> 4. Registry check is “forced” when processing a query with a function
> >> that
> >>> is not currently registered. (Doing so resolves any possible race
> >>> conditions.)
> >>>
> >>> 5. Some process (perhaps time based) removes old, unregistered jar
> files.
> >>> (Or, we could get fancy and use reference counts. The reference count
> >> would
> >>> be required if the user wants to delete, then recreate, the same
> function
> >>> and jar to avoid conflict with in-flight queries.)
> >>>
> >>> We can build security on this as follows:
> >>>
> >>> 1. Define permissions for who can write to the DFS location. Or,
> indeed,
> >>> have subdirectories by user and grant each user permission only on

Re: Time for 1.10 release

[Arina Yelchiyeva]

Hi Jinfeng,

please also consider the following Jiras (PR are already open):
* DRILL-4963: Issues when overloading Drill native functions with dynamic
UDFs
* DRILL-5255: Remove default temporary workspace check at drillbit start up
* DRILL-5274: Exception thrown in Drillbit shutdown in UDF cleanup code

Kind regards
Arina

On Thu, Feb 23, 2017 at 8:40 PM, Paul Rogers  wrote:

> Hi Jinfeng,
>
> Thanks for volunteering!
>
> The following are working their way though the system:
>
> PRs outstanding:
>
> * DRILL-5275: Sort spill is slow due to repeated allocations
> * DRILL-5260: Extend "Cluster Fixture" test framework
> * DRILL-5258: Access mock data definition from SQL
> * DRILL-5273: CompliantTextReader excessive memory use
> * DRILL-5266: Parquet returns low-density batches
> * DRILL-5257: Run-time control of query profiles
>
> The following community contribution has been approved and is ready to
> commit:
>
> * DRILL-5252: Fix a condition that always returns true
>
> The following will be (re)issued today or tomorrow:
>
> * DRILL-5114: Rationalize use of Logback logging in unit tests
> * DRILL-5284: Roll-up of final fixes for managed sort
>
> Thanks,
>
> - Paul
>
> > On Feb 23, 2017, at 9:57 AM, Sudheesh Katkam  wrote:
> >
> > I would like to include:
> >
> > + DRILL-4280: https://github.com/apache/drill/pull/578 (going through
> the last round of comments)
> >
> > Thank you,
> > Sudheesh
> >
> >> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  wrote:
> >>
> >> Hi Drillers,
> >>
> >> It has been almost 3 months since we release Drill 1.9. We have
> >> resolved plenty of fixes and improvements (closed around 88 JIRAs
> >> [1]). I propose that we start the 1.10 release process, and set
> >> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we should
> >> start build a release candidate.
> >>
> >> Please reply in this email thread if you have something near complete
> >> and you would like to include in 1.10 release.
> >>
> >> I volunteer as the release manager, unless someone else come forward.
> >>
> >> Thanks,
> >>
> >> Jinfeng
> >>
> >> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769
> >
>
>

Re: drill's calcite branch

[Arina Yelchiyeva]

Drill doesn't use Calcite directly since Drill Calcite has some commits
specific to Drill.
There are plans to migrate to the latest Calcite version [1] but there is
no exact timeline.

Kind regards
Arina

[1] https://issues.apache.org/jira/browse/DRILL-3993

On Mon, Feb 27, 2017 at 1:18 PM, weijie tong <tongweijie...@gmail.com>
wrote:

> @Arina Yelchiyeva tks for your reply.
>  what strategy does we choose to release the Calcite version? and why don't
>  we use the Calcite project itself ?
>
> On Mon, Feb 27, 2017 at 5:30 PM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
>
> > Hi,
> >
> > https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0
> >
> > Kind regards
> > Arina
> >
> > On Mon, Feb 27, 2017 at 5:51 AM, weijie tong <tongweijie...@gmail.com>
> > wrote:
> >
> > > hi Drills:
> > >   where can I find the calcite  branch source code that drill are now
> > self
> > > maintaining ?
> > >
> >
>

Re: [ANNOUNCE] New Committer: Arina Ielchiieva

[Arina Yelchiyeva]

Thank you all for congratulations! I really appreciate that.

Kind regards
Arina

On Sat, Feb 25, 2017 at 3:30 PM, Parth Chandra  wrote:

> Congratulations Arina. Welcome and thank you for your great work so far.
>
>
>
> On Fri, Feb 24, 2017 at 9:06 AM, Sudheesh Katkam 
> wrote:
>
> > The Project Management Committee (PMC) for Apache Drill has invited Arina
> > Ielchiieva to become a committer, and we are pleased to announce that she
> > has accepted.
> >
> > Arina has a long list of contributions [1] that have touched many aspects
> > of the product. Her work includes features such as dynamic UDF support
> and
> > temporary tables support.
> >
> > Welcome Arina, and thank you for your contributions.
> >
> > - Sudheesh, on behalf of the Apache Drill PMC
> >
> > [1] https://github.com/apache/drill/commits/master?author=
> arina-ielchiieva
> >
>

Re: [ANNOUNCE] New Committer: Arina Ielchiieva

[Arina Yelchiyeva]

Thank you!

On Mon, Feb 27, 2017 at 9:13 AM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> Congratulations Arina!!!
>
>       From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
>  To: dev@drill.apache.org
>  Sent: Sunday, February 26, 2017 5:23 AM
>  Subject: Re: [ANNOUNCE] New Committer: Arina Ielchiieva
>
> Thank you all for congratulations! I really appreciate that.
>
> Kind regards
> Arina
>
> On Sat, Feb 25, 2017 at 3:30 PM, Parth Chandra <par...@apache.org> wrote:
>
> > Congratulations Arina. Welcome and thank you for your great work so far.
> >
> >
> >
> > On Fri, Feb 24, 2017 at 9:06 AM, Sudheesh Katkam <sudhe...@apache.org>
> > wrote:
> >
> > > The Project Management Committee (PMC) for Apache Drill has invited
> Arina
> > > Ielchiieva to become a committer, and we are pleased to announce that
> she
> > > has accepted.
> > >
> > > Arina has a long list of contributions [1] that have touched many
> aspects
> > > of the product. Her work includes features such as dynamic UDF support
> > and
> > > temporary tables support.
> > >
> > > Welcome Arina, and thank you for your contributions.
> > >
> > > - Sudheesh, on behalf of the Apache Drill PMC
> > >
> > > [1] https://github.com/apache/drill/commits/master?author=
> > arina-ielchiieva
> > >
> >
>
>
>
>

Re: drill's calcite branch

[Arina Yelchiyeva]

Hi,

https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0

Kind regards
Arina

On Mon, Feb 27, 2017 at 5:51 AM, weijie tong 
wrote:

> hi Drills:
>   where can I find the calcite  branch source code that drill are now self
> maintaining ?
>

Re: Drill weekly meeting link

[Arina Yelchiyeva]

Apache Drill meeting is bi-weekly. It will be held next Tuesday.

Kind regards
Arina

On Tue, Sep 27, 2016 at 7:12 PM, AnilKumar B  wrote:

> I would like to join to discuss on Kafka storage plugin, but unable to
> join.
>
> Google Calendar invitations cannot be forwarded via email. This event
> belongs to drill-...@incubator.apache.org and you are logged in as
> akumarb2...@gmail.com. Please ask the meeting organizer to add you to the
> event from Google Calendar.
>
>
> Thanks & Regards,
> B Anil Kumar.
>
> On Tue, Sep 27, 2016 at 9:09 AM, AnilKumar B 
> wrote:
>
> > Hi,
> >
> > Is there any link to join Apache Drill weekly meeting?
> >
> > Thanks & Regards,
> > B Anil Kumar.
> >
>

Re: Drill Hangout will start in couple of minutes.

[Arina Yelchiyeva]

Jinfeng,

is link correct? Can't join video call.

Kind regards
Arina

On Tue, Oct 18, 2016 at 7:55 PM, Jinfeng Ni  wrote:

> Link : https://hangouts.google.com/hangouts/_/maprtech.com/
> drillbi-weeklyhangout
>
>
> Thanks,
> Jinfeng
>

Re: Drill Hangout will start in couple of minutes.

[Arina Yelchiyeva]

Unfortunately doesn't help, followed the link but only shows: Requesting to
join the video call...

Kind regards
Arina

On Tue, Oct 18, 2016 at 8:22 PM, Jinfeng Ni <j...@apache.org> wrote:

> I copied the link from Drill webpage [1].
>
> There was a connection issue in the middle of today's short hangout.
> Not sure if you just tried to join around that time.
>
>
> [1] https://drill.apache.org/community-resources/
>
> On Tue, Oct 18, 2016 at 10:06 AM, Arina Yelchiyeva
> <arina.yelchiy...@gmail.com> wrote:
> > Jinfeng,
> >
> > is link correct? Can't join video call.
> >
> > Kind regards
> > Arina
> >
> > On Tue, Oct 18, 2016 at 7:55 PM, Jinfeng Ni <j...@apache.org> wrote:
> >
> >> Link : https://hangouts.google.com/hangouts/_/maprtech.com/
> >> drillbi-weeklyhangout
> >>
> >>
> >> Thanks,
> >> Jinfeng
> >>
>

storage for Apache Design Docs

[Arina Yelchiyeva]

Hi all!

I remember a while ago there was an agreement (discussed on one of
hangouts) to create design docs for new features. But where these docs
should be kept was discussed vaguely.
Is there any place where we can keep design docs? How they can be added
there? Modified if needed?

The reason I am asking, I was going to add link to dynamic UDFs design doc
in Drill code (in comment in drill-override-example.conf) but since this
doc resides on my private Google Drive, I want to move to Apache Drill
specific storage.

Kind regards
Arina

Re: [ANNOUNCE] Apache Drill 1.9.0 Released

[Arina Yelchiyeva]

Sudheesh,

I guess the following Jiras has been included in Improvement list by
mistake, they are not in 1.9 scope:
[DRILL-4309] - Make this option
store.hive.optimize_scan_with_native_readers=true default
[DRILL-4864] - Add ANSI format for date/time functions
[DRILL-4865] - Add ANSI format for date/time functions

I guess the confusion was that they had Fix Version indicated as 1.9, it
was intended version, in reality they are still in progress / under review.
I have updated their Fix version to Future to avoid the confusion. Could
you please exclude these three Jiras from Improvement list?

Kind regards
Arina


On Wed, Nov 30, 2016 at 8:35 PM, sebb  wrote:

> What is the project about? Why should I be interested in it?
> [rhetorical questions]
>
> The Announce emails are sent to people not on the developer or user lists.
> Most will have no idea what the project is about.
>
> So the e-mails should contain at least brief details of what the
> product does, and some info on why the new release might be of
> interest to them.
>
> Readers should not have to click the link to find out the basic information
> (although of course it is useful to have such links for further detail).
>
> Please can you add that information to future announce mails?
>
> Thanks.
>
>
> On 29 November 2016 at 23:52, Sudheesh Katkam  wrote:
> > On behalf of the Apache Drill community, I am happy to announce the
> release
> > of Apache Drill 1.9.0.
> >
> > For information about Apache Drill, and to get involved, visit the
> project
> > website:
> >
> > https://drill.apache.org/
> >
> > This release introduces new features and enhancements, including
> > asynchronous Parquet reader, Parquet filter pushdown, dynamic UDF support
> > and HTTPD format plugin. In all, 70 issues have been resolved.
> >
> > The binary and source artifacts are available here:
> >
> > https://drill.apache.org/download/
> >
> > Review the release notes for a complete list of fixes and enhancements:
> >
> > https://drill.apache.org/docs/apache-drill-1-9-0-release-notes/
> >
> > Thanks to everyone in the community who contributed to this release!
> >
> > - Sudheesh
>

Re: storage for Apache Design Docs

[Arina Yelchiyeva]

Thanks, Parth!

On Mon, Dec 5, 2016 at 9:42 PM, Parth Chandra <par...@apache.org> wrote:

> I put my docs on Github (in my account) as a gist. I also put the docs on
> Google docs during the review so people could comment on them.
>
>
>
>
> On Mon, Dec 5, 2016 at 6:51 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com
> > wrote:
>
> > Hi all!
> >
> > I remember a while ago there was an agreement (discussed on one of
> > hangouts) to create design docs for new features. But where these docs
> > should be kept was discussed vaguely.
> > Is there any place where we can keep design docs? How they can be added
> > there? Modified if needed?
> >
> > The reason I am asking, I was going to add link to dynamic UDFs design
> doc
> > in Drill code (in comment in drill-override-example.conf) but since this
> > doc resides on my private Google Drive, I want to move to Apache Drill
> > specific storage.
> >
> > Kind regards
> > Arina
> >
>

Apache Drill Hangout Minutes - 12/27/16

[Arina Yelchiyeva]

Attendees: John, Vitalli, Roman, Serhii, Arina

1) John: email with NPE during select text file with options (email to user
list from Dec 8). It's hard to tell from error what is the reason of the
failure. Probably data set is needed to find the root cause.

2) Arina: when in storage plugin is indicated that text files have headers,
user can only indicated column names but not column[0], column[1] (as in
files without headers) in select statement. It would be useful to allow
both. Plus it would solve the issue with select count(1) from text file
with headers which currently results in exception (DRILL-4919).

Re: [HANGOUT] Suggestions for topics for 27/12

[Arina Yelchiyeva]

Handout is starting now.

On Mon, Dec 26, 2016 at 1:23 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Hi all,
>
> The bi-weekly hangout is tomorrow (27/12/16, 10 AM PST). If you have any 
> suggestions
> for topics for tomorrow,  please add to this thread. We will also ask for
> topics at the beginning of the hangout.
>
> Hangout link:
> *https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc
> <https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc>*
>
> Kind regards
> Arina
>

[HANGOUT] Suggestions for topics for 27/12

[Arina Yelchiyeva]

Hi all,

The bi-weekly hangout is tomorrow (27/12/16, 10 AM PST). If you have
any suggestions
for topics for tomorrow,  please add to this thread. We will also ask for
topics at the beginning of the hangout.

Hangout link:
*https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc
*

Kind regards
Arina

Hangouts: 03/21/2017

[Arina Yelchiyeva]

Will hangouts be held today?


Kind regards
Arina

Re: Temporary table issue

[Arina Yelchiyeva]

If I have understood you correctly, I don't think Drill can use Java
collection as source.
Regarding temporary table, after you create it, you can't modify it, so
yes, for every data change you would have to re-create temporary table.
One option I can think of is that you can keep your array data in file and
modify that file each time collection changes, then you can select from
this file directly or create a view over this file.

Kind regards
Arina

On Fri, Apr 7, 2017 at 9:23 AM, <jasbir.s...@accenture.com> wrote:

> Hi Arina,
>
> Thanks for your reply. I got that it can work like this also.
>
> But how to use it with Java collection, Array List which has records
> ranging from 5000 - 100,000.
> For every different hit, I will be creating a different temporary table.
>
>
> Regards,
> Jasbir Singh
>
> -Original Message-
> From: Arina Yelchiyeva [mailto:arina.yelchiy...@gmail.com]
> Sent: Wednesday, April 05, 2017 11:42 PM
> To: user <u...@drill.apache.org>
> Cc: dev@drill.apache.org
> Subject: Re: Temporary table issue
>
> HI Jasbir,
>
> may be union all and values(1) can help you.
>
> Example:
>
> create temporary table t1 as select i1, i2 from (
>
> select 0 as i1, 5 as i2 from (values(1))
>
> union all
>
> select 5 as i1, 10 as i2 from (values(1)));
>
>
> Kind regards
>
> Arina
>
> On Wed, Apr 5, 2017 at 7:31 PM, <jasbir.s...@accenture.com> wrote:
>
> > Hi,
> >
> > I want to create temporary table in Apache Drill at runtime and want
> > to insert few uuid's which user select. It could range from 5000 to
> 100,000.
> > Using CTTAS if I do it requires data from some file whereas here I
> > want to store it using a collection.
> >
> > Is this possible using Apache Drill?
> >
> > Regards,
> > Jasbir Singh
> >
> > 
> >
> > This message is for the designated recipient only and may contain
> > privileged, proprietary, or otherwise confidential information. If you
> > have received it in error, please notify the sender immediately and
> > delete the original. Any other use of the e-mail by you is prohibited.
> > Where allowed by local law, electronic communications with Accenture
> > and its affiliates, including e-mail and instant messaging (including
> > content), may be scanned by our systems for the purposes of
> > information security and assessment of internal compliance with
> Accenture policy.
> > 
> > __
> >
> > www.accenture.com
> >
>
> 
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> 
> __
>
> www.accenture.com
>

Re: Temporary table issue

[Arina Yelchiyeva]

HI Jasbir,

may be union all and values(1) can help you.

Example:

create temporary table t1 as select i1, i2 from (

select 0 as i1, 5 as i2 from (values(1))

union all

select 5 as i1, 10 as i2 from (values(1)));


Kind regards

Arina

On Wed, Apr 5, 2017 at 7:31 PM,  wrote:

> Hi,
>
> I want to create temporary table in Apache Drill at runtime and want to
> insert few uuid's which user select. It could range from 5000 to 100,000.
> Using CTTAS if I do it requires data from some file whereas here I want to
> store it using a collection.
>
> Is this possible using Apache Drill?
>
> Regards,
> Jasbir Singh
>
> 
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> 
> __
>
> www.accenture.com
>

Re: [VOTE] Release Apache Drill 1.10.0 - rc0

[Arina Yelchiyeva]

So far found one issue:

https://issues.apache.org/jira/browse/DRILL-5338


Kind regards
Arina

On Thu, Mar 9, 2017 at 9:31 AM, Jinfeng Ni  wrote:

> Hello all,
>
> I'd like to propose the first release candidate (rc0) of Apache Drill,
> version 1.10.0.
>
> The release candidate covers a total of 126 resolved JIRAs [1]. Thanks
> to everyone who contributed to this release.
>
> The tarball artifacts are hosted at [2] and the maven artifacts are
> hosted at [3].
>
> This release candidate is based on commit
> 38ef562b1ced59efe57b0dc606f2c36694569102 located at [4].
>
> The vote will be open for the next ~96 hours (including an extra day
> as the vote is happening over a weekend) ending at 8:00AM Pacific,
> March 13th, 2017.
>
> [ ] +1
> [ ] +0
> [ ] -1
>
> Here's my vote: +1
>
> Thanks,
>
> Jinfeng
>
> 1. https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12338769=12313820
> 2. http://home.apache.org/~jni/drill/1.10.0/rc0/
> 3. https://repository.apache.org/content/repositories/orgapachedrill-1041/
> 4. https://github.com/jinfengni/incubator-drill/commits/drill-1.10.0
>

Re: [VOTE] Release Apache Drill 1.10.0 - rc0

[Arina Yelchiyeva]

+1

- Downloaded tarball and started drill in embedded mode on Windows. Run
simple queries, checked CTTAS and dynamic UDFs features.
- Downloaded from the source on Linux VM and did build with unit tests.
- Deployed on 4 node cluster, run tests from test framework, executed
simple queries.

Kind regards
Arina

On Sat, Mar 11, 2017 at 4:09 AM, Aman Sinha <asi...@mapr.com> wrote:

> Never mind. Release note link seems fine.
>
> On 3/10/17, 6:06 PM, "Aman Sinha" <asi...@mapr.com> wrote:
>
> +1 (binding)
>
> - Downloaded the binaries on my mac, checked the git.properties and
> KEYS file (for PGP key)
> - Started Drill in embedded mode and ran a few sample queries with and
> without metadata cache.  Checked query profiles.
>  -Tested query cancellation for couple of queries
> - Downloaded source on my Linux VM, did a build and ran unit tests
> successfully.
> - Checked Maven artifacts on repository.apache.org
>
> Release notes link in [1] seems broken. Can you provide the correct
> link ?
>
> -Aman
>
> On 3/10/17, 11:44 AM, "Jinfeng Ni" <j...@apache.org> wrote:
>
> Agreed that DRILL-5338 is not a blocker for the release.
>
> Just a friendly reminder. Vote ends on Monday 8:00AM. Please try
> out
> RC0, and cast your vote before it ends.
>
> Thanks,
>
> Jinfeng
>
>
> On Thu, Mar 9, 2017 at 9:53 AM, Paul Rogers <prog...@mapr.com>
> wrote:
> > Added a note to the JIRA.
> >
> > Yes, I had asked Kunal to provide some visual indication that
> the detail values are different parts of the total run time. Neither of us
> are UI designers, so if anyone has a better visual solution, please let us
> know!
> >
> > This should not be a blocker.
> >
> > Thanks,
> >
> > - Paul
> >
> >> On Mar 9, 2017, at 8:30 AM, Zelaine Fong <zf...@mapr.com>
> wrote:
> >>
> >> This looks like it’s due to the changes Kunal made in
> https://issues.apache.org/jira/browse/DRILL-5190.  Looking at the code
> changes, it appears that the indentation was intended, as it denotes that
> those indented timings are a breakdown of the overall duration.
> >>
> >> Kunal?
> >>
> >> -- Zelaine
> >>
> >> On 3/9/17, 8:20 AM, "Arina Yelchiyeva" <
> arina.yelchiy...@gmail.com> wrote:
> >>
> >>So far found one issue:
> >>
> >>https://issues.apache.org/jira/browse/DRILL-5338
> >>
> >>
> >>Kind regards
> >>Arina
> >>
> >>On Thu, Mar 9, 2017 at 9:31 AM, Jinfeng Ni <j...@apache.org>
> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to propose the first release candidate (rc0) of
> Apache Drill,
> >>> version 1.10.0.
> >>>
> >>> The release candidate covers a total of 126 resolved JIRAs
> [1]. Thanks
> >>> to everyone who contributed to this release.
> >>>
> >>> The tarball artifacts are hosted at [2] and the maven
> artifacts are
> >>> hosted at [3].
> >>>
> >>> This release candidate is based on commit
> >>> 38ef562b1ced59efe57b0dc606f2c36694569102 located at [4].
> >>>
> >>> The vote will be open for the next ~96 hours (including an
> extra day
> >>> as the vote is happening over a weekend) ending at 8:00AM
> Pacific,
> >>> March 13th, 2017.
> >>>
> >>> [ ] +1
> >>> [ ] +0
> >>> [ ] -1
> >>>
> >>> Here's my vote: +1
> >>>
> >>> Thanks,
> >>>
> >>> Jinfeng
> >>>
> >>> 1. https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >>> version=12338769=12313820
> >>> 2. http://home.apache.org/~jni/drill/1.10.0/rc0/
> >>> 3. https://repository.apache.org/content/repositories/
> orgapachedrill-1041/
> >>> 4. https://github.com/jinfengni/incubator-drill/commits/drill-
> 1.10.0
> >>>
> >>
> >>
> >
>
>
>
>
>

[RESULT] [VOTE] Release Apache Drill 1.11.0 rc0

[Arina Yelchiyeva]

The vote passes. Thanks to everyone who has tested the release candidate
and given their comments and votes. Final tally:

3x +1 (binding): Parth, Jinfeng, Aman

6x +1 (non-binding): Arina, Kunal, Padma, Abhishek, Sorabh, Khurram

No 0s or -1s.

I'll start the process for pushing the release artifacts and send an
announcement once propagated.

Kind regards
Arina

[ANNOUNCE] Apache Drill 1.11.0 Released

[Arina Yelchiyeva]

On behalf of the Apache Drill community, I am happy to announce the release
of Apache Drill 1.11.0.

For information about Apache Drill, and to get involved, visit the project
website [1].

This release of Drill provides the following new features and improvements
[2]:

Cryptography-related functions. (DRILL-5634)
Spill to disk for the hash aggregate operator. (DRILL-5457)
Format plugin support for PCAP files. (DRILL-5432)
Ability to change the HDFS block Size for Parquet files. (DRILL-5379)
Ability to store query profiles in memory. (DRILL-5481)
Configurable CTAS directory and file permissions option. (DRILL-5391)
Support for network encryption. (DRILL-4335)
Relative paths stored in the metadata file. (DRILL-3867)
Support for ANSI_QUOTES. (DRILL-3510)

The binary and source artifacts are available here [3].

Thanks to everyone in the community who contributed to this release!

1. https://drill.apache.org/
2. https://drill.apache.org/docs/apache-drill-1-11-0-release-notes/
3. https://drill.apache.org/download/

Kind regards
Arina

Re: [ANNOUNCE] New PMC member: Arina Ielchiieva

[Arina Yelchiyeva]

Thank all you!

Kind regards
Arina

On Thu, Aug 3, 2017 at 5:58 AM, Sudheesh Katkam  wrote:

> Congratulations and thank you, Arina.
>
> On Wed, Aug 2, 2017 at 1:38 PM, Paul Rogers  wrote:
>
> > The success of the Drill 1.11 release proves this is a well-deserved
> move.
> > Congratulations!
> >
> > - Paul
> >
> > > On Aug 2, 2017, at 11:23 AM, Aman Sinha  wrote:
> > >
> > > I am pleased to announce that Drill PMC invited Arina Ielchiieva to the
> > PMC
> > > and she has accepted the invitation.
> > >
> > > Congratulations Arina and thanks for your contributions !
> > >
> > > -Aman
> > > (on behalf of Drill PMC)
> >
> >
>

Re: Why Drill required a special Calcite fork ?

[Arina Yelchiyeva]

Hi Muhammad,

Link - https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0

Kind regards
Arina

On Mon, Jul 17, 2017 at 6:52 PM, Muhammad Gelbana 
wrote:

> Could anyone at least just provide a link for that fork's repository please
> ?
>
>
>
> - Gelbana
>
> On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana 
> wrote:
>
> > Would someone please list the reasons for which Drill required having a
> > custom Calcite build ? This will help integrating Calcite whenever a
> > release is out.
> >
> > One reason I can assume is that Calcite didn't support unparsing all SQL
> > clauses such as FETCH and OFFSET ?
> >
> > I for my self, very much need to use the latest Calcite version with
> Drill
> > and I'm willing to spend time working on that, if it's possible for my
> poor
> > Drill and Calcite knowledge.
> >
> > ​-
> > Gelbana
> >
>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

@Jinfeng,
Let's discuss today on Drill hangout final cut-off date.

@Charles
Since there were questions about AES in DRILL-5634, should we exclude it
from the candidates for 1.11.0 release?

@Boaz
Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code review?

Also I believe we should include DRILL-5660 where we need to bump up
parquet metadata version.
@Vitalii
Do you have any ETA for the fix?

Kind regards
Arina

On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bben-...@mapr.com> wrote:

>  Hi Arina,
>
>  Two PR have just been submitted, relating to the Hash Aggr Spill:
> DRILL-5616 and DRILL-5665 .
> Can these be considered for 1.11 as well ?
>
>  Thanks,
>
>Boaz
>
> On 7/10/17, 1:49 PM, "Paul Rogers" <prog...@mapr.com> wrote:
>
> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
> after the release to allow others more time to work on other features
> rather than doing code reviews.
>
> - Paul
>
> > On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <j...@apache.org> wrote:
> >
> > Hi Arina,
> >
> > It has been a while since we discussed 1.11.0 release.  Any update
> on the
> > new release plan?  Do we have a cut-off date ?
> >
> > In the past, I believe Drill release is a time-bounded, not feature
> > bounded. If certain fixes/features could not make it within a certain
> > time-frame, we had better defer them into future release, unless
> those
> > fixes/features are regarded as blocking issues for a new release.
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com
> >> wrote:
> >
> >> Latest update:
> >>
> >> We wait for the following Jiras before final cut off.
> >> 1. DRILL-5601  - Paul
> >> 2. DRILL-5420 - Kunal
> >> 3. DRILL-5634 - Charles
> >>
> >> Kind regards
> >> Arina
> >>
> >> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kkha...@mapr.com>
> wrote:
> >>
> >>> Parth's already +1'd the process with a minor update. So, we're
> good to
> >> go
> >>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> >>>
> >>> <http://www.mapr.com/>
> >>>
> >>> 
>     >>> From: Parth Chandra <par...@apache.org>
> >>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>> To: dev@drill.apache.org
> >>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>
> >>> I'm Ok with the pcap format plugin if Pauls comments are also
> addressed.
> >> If
> >>> the changes are done before your cutoff deadline,we can merge it
> in.
> >>>
> >>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>> arina.yelchiy...@gmail.com> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> here is the list of Jira we are waiting before making final cut
> off:
>     >>>> 1. DRILL-5601  - Paul
> >>>> 2. DRILL-3640  - Kunal
> >>>> 3. DRILL-5420 - Kunal
> >>>>
> >>>> Please let me know if there are any others Jiras we should wait
> for.
> >>>>
> >>>> @Charles,
> >>>> you have mentioned that you want to include some UDFs in Drill
> 1.11, do
> >>> you
> >>>> have any ETA for the PR?
> >>>>
> >>>> @Paul / Parth
> >>>> I see there is active CR for pcap format PR, do we want to
> include it
> >> in
> >>>> this release as well?
> >>>>
> >>>> Kind regards
> >>>> Arina
> >>>>
> >>>>
> >>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>> arina.yelchiy...@gmail.com> wrote:
> >>>>
> >>>>> Parth, thanks a lot the instructions.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> par...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> FW

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

During today hangouts we agreed on approximate cut-off date - 14 July, 2017
(date may be shifted due to the blockers).
I'll start preparing test RC on Thursday to solve any release preparation
issues beforehand.

*Blockers*
DRILL-5660 - Vitalii

*Possible blockers*
DRILL-5468 - Jinfeng
NPE on CSV source - Paul
DRILL-5659 - Parth

*Will be included if done before the cut-off date*
DRILL-5616 / DRILL-5665 - Boaz
DRILL-5634 - Parth will take a look at Apache commons implementation and
update.

Kind regards
Arina

On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cgi...@gmail.com> wrote:

> Hi Arina,
> I can’t make the hangout, but we have a few options.  Basically, since all
> the hashing algorithms have no export restrictions, we could:
> 1.  Remove the AES encrypt/decrypt until I can figure out what we have to
> do (and do it)
> 2.  I can try to research it today or tomorrow to figure out exactly what
> we have to do to include AES.
> 3.  Remove the entire package
>
> I’d like to see the hashing functions at least included because we’ve had
> a lot of requests for that functionality.   I also don’t want this to hold
> things up, so please let me know your preference.  Regardless, I’m going to
> start trying to figure out the restrictions so we’ll know for the future.
>
> Best,
> — C
>
>
>
> > On Jul 11, 2017, at 05:58, Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> wrote:
> >
> > @Jinfeng,
> > Let's discuss today on Drill hangout final cut-off date.
> >
> > @Charles
> > Since there were questions about AES in DRILL-5634, should we exclude it
> > from the candidates for 1.11.0 release?
> >
> > @Boaz
> > Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code review?
> >
> > Also I believe we should include DRILL-5660 where we need to bump up
> > parquet metadata version.
> > @Vitalii
> > Do you have any ETA for the fix?
> >
> > Kind regards
> > Arina
> >
> > On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bben-...@mapr.com> wrote:
> >
> >> Hi Arina,
> >>
> >> Two PR have just been submitted, relating to the Hash Aggr Spill:
> >> DRILL-5616 and DRILL-5665 .
> >> Can these be considered for 1.11 as well ?
> >>
> >> Thanks,
> >>
> >>   Boaz
> >>
> >> On 7/10/17, 1:49 PM, "Paul Rogers" <prog...@mapr.com> wrote:
> >>
> >>DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
> >> after the release to allow others more time to work on other features
> >> rather than doing code reviews.
> >>
> >>- Paul
> >>
> >>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <j...@apache.org> wrote:
> >>>
> >>> Hi Arina,
> >>>
> >>> It has been a while since we discussed 1.11.0 release.  Any update
> >> on the
> >>> new release plan?  Do we have a cut-off date ?
> >>>
> >>> In the past, I believe Drill release is a time-bounded, not feature
> >>> bounded. If certain fixes/features could not make it within a certain
> >>> time-frame, we had better defer them into future release, unless
> >> those
> >>> fixes/features are regarded as blocking issues for a new release.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >> arina.yelchiy...@gmail.com
> >>>> wrote:
> >>>
> >>>> Latest update:
> >>>>
> >>>> We wait for the following Jiras before final cut off.
> >>>> 1. DRILL-5601  - Paul
> >>>> 2. DRILL-5420 - Kunal
> >>>> 3. DRILL-5634 - Charles
> >>>>
> >>>> Kind regards
> >>>> Arina
> >>>>
> >>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kkha...@mapr.com>
> >> wrote:
> >>>>
> >>>>> Parth's already +1'd the process with a minor update. So, we're
> >> good to
> >>>> go
> >>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> >>>>>
> >>>>> <http://www.mapr.com/>
> >>>>>
> >>>>> 
> >>>>> From: Parth Chandra <par...@apache.org>
> >>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>> To: dev@drill.apache.org
> >>>>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

*Blockers*
DRILL-5659 - problem is identified, Parth is working on the fix.

*Not blockers*
DRILL-5660 - Vitalii
DRILL-5468 - Jinfeng

*Possible blockers*
DRILL-5669 - Boaz Ben-Zvi / Paul
NPE on CSV - Paul

Kind regards
Arina


On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <prog...@mapr.com> wrote:

> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
> track down the incompatibility the issue internally, but production users
> probably don’t attempt to use one build of Drill with metadata files
> created from another. We can worry about the issue later only if a user
> actually runs into it.
>
> As suggested earlier, let’s just document this obscure case in a release
> note.
>
> - Paul
>
> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <j...@apache.org> wrote:
> >
> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> cache
> > file is involved.
> >
> > I might be wrong, but I thought that if someone is running on 1.10.0, and
> > he wants to go back to 1.0.0, where the first version of parquet
> meatadata
> > cache was introduced, can we guarantee that Drillbit would work
> correctly?
> > I'm trying to understand what DRILL-5660 asks for is a new requirement,
> or
> > an requirement being there starting from Drill 1.0.0?
> >
> > Regression failures are normally regarded as release blocker.
> >
> >
> >
> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <prog...@mapr.com> wrote:
> >
> >> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all,
> so
> >> that is a time savings. We could handle this via a release note item
> that
> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
> not
> >> support the ability to fall back to the old version. Everyone OK with
> that?
> >>
> >> We are seeing some stress test regression failures that, I’m told, will
> >> show up as JIRA entries shortly.
> >>
> >> - Paul
> >>
> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <j...@apache.org> wrote:
> >>>
> >>> I put some analysis in DRILL-5468.
> >>>
> >>> Also, IMHO, DRILL-5660 is not a release blocker.
> >>>
> >>>
> >>>
> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <par...@apache.org>
> >> wrote:
> >>>
> >>>> Re DRILL-5634:
> >>>> From the Apache commons web page [1] -
> >>>>
> >>>> Please note that Apache Commons Crypto doesn't implement the
> >> cryptographic
> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
> >> implement
> >>>> the algorithms.*
> >>>>
> >>>>
> >>>> Since we are cleared for using Openssl and JCE, we are OK to include
> the
> >>>> AES code in the UDFs.
> >>>>
> >>>>
> >>>> Parth
> >>>>
> >>>> [1] https://commons.apache.org/proper/commons-crypto/
> >>>>
> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> >>>> arina.yelchiy...@gmail.com> wrote:
> >>>>
> >>>>> During today hangouts we agreed on approximate cut-off date - 14
> July,
> >>>> 2017
> >>>>> (date may be shifted due to the blockers).
> >>>>> I'll start preparing test RC on Thursday to solve any release
> >> preparation
> >>>>> issues beforehand.
> >>>>>
> >>>>> *Blockers*
> >>>>> DRILL-5660 - Vitalii
> >>>>>
> >>>>> *Possible blockers*
> >>>>> DRILL-5468 - Jinfeng
> >>>>> NPE on CSV source - Paul
> >>>>> DRILL-5659 - Parth
> >>>>>
> >>>>> *Will be included if done before the cut-off date*
> >>>>> DRILL-5616 / DRILL-5665 - Boaz
> >>>>> DRILL-5634 - Parth will take a look at Apache commons implementation
> >> and
> >>>>> update.
> >>>>>
> >>>>> Kind regards
> >>>>> Arina
> >>>>>
> >>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cgi...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi Arina,
> >>>>>> I can’t make the hangout, but we have a few options.  Basically,
> since
> >>>>> all
> >>>>>> the

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

*Blockers*
DRILL-5659 - Parth (done)
DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)

*Possible blockers*
NPE on CSV - Paul can you please provide an update?

Kind regards
Arina

On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> *Blockers*
> DRILL-5659 - problem is identified, Parth is working on the fix.
>
> *Not blockers*
> DRILL-5660 - Vitalii
> DRILL-5468 - Jinfeng
>
> *Possible blockers*
> DRILL-5669 - Boaz Ben-Zvi / Paul
> NPE on CSV - Paul
>
> Kind regards
> Arina
>
>
> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <prog...@mapr.com> wrote:
>
>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
>> track down the incompatibility the issue internally, but production users
>> probably don’t attempt to use one build of Drill with metadata files
>> created from another. We can worry about the issue later only if a user
>> actually runs into it.
>>
>> As suggested earlier, let’s just document this obscure case in a release
>> note.
>>
>> - Paul
>>
>> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <j...@apache.org> wrote:
>> >
>> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
>> cache
>> > file is involved.
>> >
>> > I might be wrong, but I thought that if someone is running on 1.10.0,
>> and
>> > he wants to go back to 1.0.0, where the first version of parquet
>> meatadata
>> > cache was introduced, can we guarantee that Drillbit would work
>> correctly?
>> > I'm trying to understand what DRILL-5660 asks for is a new requirement,
>> or
>> > an requirement being there starting from Drill 1.0.0?
>> >
>> > Regression failures are normally regarded as release blocker.
>> >
>> >
>> >
>> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <prog...@mapr.com> wrote:
>> >
>> >> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all,
>> so
>> >> that is a time savings. We could handle this via a release note item
>> that
>> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
>> not
>> >> support the ability to fall back to the old version. Everyone OK with
>> that?
>> >>
>> >> We are seeing some stress test regression failures that, I’m told, will
>> >> show up as JIRA entries shortly.
>> >>
>> >> - Paul
>> >>
>> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <j...@apache.org> wrote:
>> >>>
>> >>> I put some analysis in DRILL-5468.
>> >>>
>> >>> Also, IMHO, DRILL-5660 is not a release blocker.
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <par...@apache.org>
>> >> wrote:
>> >>>
>> >>>> Re DRILL-5634:
>> >>>> From the Apache commons web page [1] -
>> >>>>
>> >>>> Please note that Apache Commons Crypto doesn't implement the
>> >> cryptographic
>> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
>> >> implement
>> >>>> the algorithms.*
>> >>>>
>> >>>>
>> >>>> Since we are cleared for using Openssl and JCE, we are OK to include
>> the
>> >>>> AES code in the UDFs.
>> >>>>
>> >>>>
>> >>>> Parth
>> >>>>
>> >>>> [1] https://commons.apache.org/proper/commons-crypto/
>> >>>>
>> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>> >>>> arina.yelchiy...@gmail.com> wrote:
>> >>>>
>> >>>>> During today hangouts we agreed on approximate cut-off date - 14
>> July,
>> >>>> 2017
>> >>>>> (date may be shifted due to the blockers).
>> >>>>> I'll start preparing test RC on Thursday to solve any release
>> >> preparation
>> >>>>> issues beforehand.
>> >>>>>
>> >>>>> *Blockers*
>> >>>>> DRILL-5660 - Vitalii
>> >>>>>
>> >>>>> *Possible blockers*
>> >>>>> DRILL-5468 - Jinfeng
>> >>>>> NPE on CSV source - Paul
>> >>>>> DRILL-5659 - Parth
>> >>>>>
>&

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Hi Parth,

no problem, I did not start preparing the release since there is one
blocker has left:

*DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*


Kind regards
Arina

On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <par...@apache.org> wrote:

> Hi Arina,
>
>   I just realized that I did not push the C++ client fix ( DRILL-5659) to
> master. I have just done so.
>
>   Sorry if I messed things up for the release, but we should get this fix
> in.
>
> Parth
>
> On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <prog...@mapr.com> wrote:
>
> > I looked into one NPE on CSV. Turns out it is already fixed by previous
> > 1.11 PRs, so we’re good.
> >
> > There is another reported by a user on the user list, but I’ve not heard
> > back about getting sample data. Let’s not hold the release for that.
> >
> > - Paul
> >
> > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> > >
> > > *Blockers*
> > > DRILL-5659 - Parth (done)
> > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> > >
> > > *Possible blockers*
> > > NPE on CSV - Paul can you please provide an update?
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> > > arina.yelchiy...@gmail.com> wrote:
> > >
> > >> *Blockers*
> > >> DRILL-5659 - problem is identified, Parth is working on the fix.
> > >>
> > >> *Not blockers*
> > >> DRILL-5660 - Vitalii
> > >> DRILL-5468 - Jinfeng
> > >>
> > >> *Possible blockers*
> > >> DRILL-5669 - Boaz Ben-Zvi / Paul
> > >> NPE on CSV - Paul
> > >>
> > >> Kind regards
> > >> Arina
> > >>
> > >>
> > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <prog...@mapr.com>
> wrote:
> > >>
> > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us
> to
> > >>> track down the incompatibility the issue internally, but production
> > users
> > >>> probably don’t attempt to use one build of Drill with metadata files
> > >>> created from another. We can worry about the issue later only if a
> user
> > >>> actually runs into it.
> > >>>
> > >>> As suggested earlier, let’s just document this obscure case in a
> > release
> > >>> note.
> > >>>
> > >>> - Paul
> > >>>
> > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <j...@apache.org> wrote:
> > >>>>
> > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> > >>> cache
> > >>>> file is involved.
> > >>>>
> > >>>> I might be wrong, but I thought that if someone is running on
> 1.10.0,
> > >>> and
> > >>>> he wants to go back to 1.0.0, where the first version of parquet
> > >>> meatadata
> > >>>> cache was introduced, can we guarantee that Drillbit would work
> > >>> correctly?
> > >>>> I'm trying to understand what DRILL-5660 asks for is a new
> > requirement,
> > >>> or
> > >>>> an requirement being there starting from Drill 1.0.0?
> > >>>>
> > >>>> Regression failures are normally regarded as release blocker.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <prog...@mapr.com>
> > wrote:
> > >>>>
> > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at
> > all,
> > >>> so
> > >>>>> that is a time savings. We could handle this via a release note
> item
> > >>> that
> > >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we
> > do
> > >>> not
> > >>>>> support the ability to fall back to the old version. Everyone OK
> with
> > >>> that?
> > >>>>>
> > >>>>> We are seeing some stress test regression failures that, I’m told,
> > will
> > >>>>> show up as JIRA entries shortly.
> > >>>>>
> > >>>>> - Paul
> > >>>>>
> > >>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Thanks, Boaz!

Unfortunately, I still see the failures connected to DRILL-5669.
I'll share the details.

Kind regards
Arina

On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <bben-...@mapr.com> wrote:

> Hi Arina,
>
>  The three Jira’s are ready (+1, squashed, rebased, etc.):
>
> • DRILL-5616: Add memory checks, plus minor metrics changes
> (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
> • DRILL-5665: Add the option planner.force_2phase_aggr to override small
> inputs
> (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
> • DRILL-5669: Add a configurable option for minimum memory allocation to
> buffered ops
> (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )
>
>   Thanks,
>
>      Boaz
>
> On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <arina.yelchiy...@gmail.com>
> wrote:
>
> Hi Parth,
>
> no problem, I did not start preparing the release since there is one
> blocker has left:
>
> *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
>
>
> Kind regards
> Arina
>
> On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <par...@apache.org>
> wrote:
>
> > Hi Arina,
> >
> >   I just realized that I did not push the C++ client fix (
> DRILL-5659) to
> > master. I have just done so.
> >
> >   Sorry if I messed things up for the release, but we should get
> this fix
> > in.
> >
> > Parth
> >
> > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <prog...@mapr.com>
> wrote:
> >
> > > I looked into one NPE on CSV. Turns out it is already fixed by
> previous
> > > 1.11 PRs, so we’re good.
> > >
> > > There is another reported by a user on the user list, but I’ve not
> heard
> > > back about getting sample data. Let’s not hold the release for
> that.
> > >
> > > - Paul
> > >
> > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
> > > arina.yelchiy...@gmail.com> wrote:
> > > >
> > > > *Blockers*
> > > > DRILL-5659 - Parth (done)
> > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> > > >
> > > > *Possible blockers*
> > > > NPE on CSV - Paul can you please provide an update?
> > > >
> > > > Kind regards
> > > > Arina
> > > >
> > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> > > > arina.yelchiy...@gmail.com> wrote:
> > > >
> > > >> *Blockers*
> > > >> DRILL-5659 - problem is identified, Parth is working on the fix.
> > > >>
> > > >> *Not blockers*
> > > >> DRILL-5660 - Vitalii
> > > >> DRILL-5468 - Jinfeng
> > > >>
> > > >> *Possible blockers*
> > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
> > > >> NPE on CSV - Paul
> > > >>
> > > >> Kind regards
> > > >> Arina
> > > >>
> > > >>
> > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <prog...@mapr.com>
> > wrote:
> > > >>
> > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle
> for us
> > to
> > > >>> track down the incompatibility the issue internally, but
> production
> > > users
> > > >>> probably don’t attempt to use one build of Drill with metadata
> files
> > > >>> created from another. We can worry about the issue later only
> if a
> > user
> > > >>> actually runs into it.
> > > >>>
> > > >>> As suggested earlier, let’s just document this obscure case in
> a
> > > release
> > > >>> note.
> > > >>>
> > > >>> - Paul
> > > >>>
> > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <j...@apache.org>
> wrote:
> > > >>>>
> > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet
> metadata
> > > >>> cache
> > > >>>> file is involved.
> > > >>>>
> > > >>>> I migh

[VOTE] Release Apache Drill 1.11.0 - rc0

[Arina Yelchiyeva]

Hi all,

I'd like to propose the first release candidate (rc0) of Apache Drill, version
1.11.0.

The release candidate covers a total of 126 resolved JIRAs [1]. Thanks to
everyone who contributed to this release.

The tarball artifacts are hosted at [2] and the maven artifacts are hosted
at [3].

This release candidate is based on commit
4220fb2fffbc81883df3e5fea575fa0a584852b3 located at [4].

The vote ends at 1:00 PM UTC (5:00 AM PT), July 28, 2017.

[ ] +1
[ ] +0
[ ] -1

Here's my vote: +1 (non-binding)


[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12339943
[2] http://home.apache.org/~arina/drill/releases/1.11.0/rc0/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1042/
[4] https://github.com/arina-ielchiieva/drill/commits/drill-1.11.0

Kind regards
Arina

Re: [HANGOUT] Topics for 7/25/17

[Arina Yelchiyeva]

Hangouts is starting now...

On Tue, Jul 25, 2017 at 7:41 AM, Padma Penumarthy <ppenumar...@mapr.com>
wrote:

> I have a topic to discuss. Lot of folks on the user mailing list raised
> the issue of not being able to access all S3 regions using Drill.
> We need hadoop version 2.8 or higher to be able to connect to
> regions which support only Version 4 signature.
> I tried with 2.8.1, which just got released and it works i.e. I am able to
> connect to both old and new regions (by specifying the endpoint in the
> config).
> There are some failures in unit tests, which can be fixed.
>
> Fixing S3 connectivity issues is important.
> However, the hadoop release notes for 2.8.1 (and 2.8.0 as well) say the
> following:
> "Please note that 2.8.x release line continues to be not yet ready for
> production use”.
>
> So, should we or not move to 2.8.1 ?
>
> Thanks,
> Padma
>
>
> On Jul 24, 2017, at 9:46 AM, Arina Yelchiyeva <arina.yelchiy...@gmail.com<
> mailto:arina.yelchiy...@gmail.com>> wrote:
>
> Hi all,
>
> We'll have the hangout tomorrow at the usual time [1]. Any topics to be
> discussed?
>
> [1] https://drill.apache.org/community-resources/
>
> Kind regards
> Arina
>
>

Re: [HANGOUT] Topics for 7/25/17

[Arina Yelchiyeva]

Meeting minutes 25 July 2017:

Attendees:
Rob, Vova, Sorabh, Pritesh, Paul, Aman, Padma, Jyothsna, Sindhuri.

Two topics were discussed.
1. Release candidate for 1.11.0.
Everybody is encouraged to test the release candidate and vote.
Aman asked about the release candidate performance testing.
Asked Kunal via email and he confirmed that performance testing is in
progress.

2. Upgrade to hadoop version 2.8.
Padma was looking into S3 connectivity issues and found out that switching
to Hadoop version 2.8.1 will solve these problems.
However, the hadoop release notes for 2.8.1 (and 2.8.0 as well) say the
following:
"Please note that 2.8.x release line continues to be not yet ready for
production use”.
Was decided to wait till the next Hadoop stable version release (hopefully
before Drill 1.12.0 release)
and for now document that users may switch to 2.8.1 themselves.


Thank you all for attending the hangout today.

Kind regards
Arina

On Tue, Jul 25, 2017 at 8:04 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Hangouts is starting now...
>
> On Tue, Jul 25, 2017 at 7:41 AM, Padma Penumarthy <ppenumar...@mapr.com>
> wrote:
>
>> I have a topic to discuss. Lot of folks on the user mailing list raised
>> the issue of not being able to access all S3 regions using Drill.
>> We need hadoop version 2.8 or higher to be able to connect to
>> regions which support only Version 4 signature.
>> I tried with 2.8.1, which just got released and it works i.e. I am able to
>> connect to both old and new regions (by specifying the endpoint in the
>> config).
>> There are some failures in unit tests, which can be fixed.
>>
>> Fixing S3 connectivity issues is important.
>> However, the hadoop release notes for 2.8.1 (and 2.8.0 as well) say the
>> following:
>> "Please note that 2.8.x release line continues to be not yet ready for
>> production use”.
>>
>> So, should we or not move to 2.8.1 ?
>>
>> Thanks,
>> Padma
>>
>>
>> On Jul 24, 2017, at 9:46 AM, Arina Yelchiyeva <arina.yelchiy...@gmail.com
>> <mailto:arina.yelchiy...@gmail.com>> wrote:
>>
>> Hi all,
>>
>> We'll have the hangout tomorrow at the usual time [1]. Any topics to be
>> discussed?
>>
>> [1] https://drill.apache.org/community-resources/
>>
>> Kind regards
>> Arina
>>
>>
>

[HANGOUT] Topics for 7/25/17

[Arina Yelchiyeva]

Hi all,

We'll have the hangout tomorrow at the usual time [1]. Any topics to be
discussed?

[1] https://drill.apache.org/community-resources/

Kind regards
Arina

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

I have merged all pending Jiras (including blockers) and will start building
RC0 shortly.

*Note for the committers:*
*until the release is not over and Drill version is not changed to
1.12.0-SNAPSHOT, please do not push any changes into Drill master. *

Kind regards
Arina

On Wed, Jul 19, 2017 at 12:15 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Thanks, Boaz!
>
> Unfortunately, I still see the failures connected to DRILL-5669.
> I'll share the details.
>
> Kind regards
> Arina
>
> On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <bben-...@mapr.com> wrote:
>
>> Hi Arina,
>>
>>  The three Jira’s are ready (+1, squashed, rebased, etc.):
>>
>> • DRILL-5616: Add memory checks, plus minor metrics changes
>> (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
>> • DRILL-5665: Add the option planner.force_2phase_aggr to override small
>> inputs
>> (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
>> • DRILL-5669: Add a configurable option for minimum memory allocation to
>> buffered ops
>> (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )
>>
>>   Thanks,
>>
>>  Boaz
>>
>> On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <arina.yelchiy...@gmail.com>
>> wrote:
>>
>> Hi Parth,
>>
>> no problem, I did not start preparing the release since there is one
>> blocker has left:
>>
>> *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
>>
>>
>> Kind regards
>> Arina
>>
>> On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <par...@apache.org>
>> wrote:
>>
>> > Hi Arina,
>> >
>> >   I just realized that I did not push the C++ client fix (
>> DRILL-5659) to
>> > master. I have just done so.
>> >
>> >   Sorry if I messed things up for the release, but we should get
>> this fix
>> > in.
>> >
>> > Parth
>> >
>> > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <prog...@mapr.com>
>> wrote:
>> >
>> > > I looked into one NPE on CSV. Turns out it is already fixed by
>> previous
>> > > 1.11 PRs, so we’re good.
>> > >
>> > > There is another reported by a user on the user list, but I’ve
>> not heard
>> > > back about getting sample data. Let’s not hold the release for
>> that.
>> > >
>> > > - Paul
>> > >
>> > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
>> > > arina.yelchiy...@gmail.com> wrote:
>> > > >
>> > > > *Blockers*
>> > > > DRILL-5659 - Parth (done)
>> > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of
>> days)
>> > > >
>> > > > *Possible blockers*
>> > > > NPE on CSV - Paul can you please provide an update?
>> > > >
>> > > > Kind regards
>> > > > Arina
>> > > >
>> > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
>> > > > arina.yelchiy...@gmail.com> wrote:
>> > > >
>> > > >> *Blockers*
>> > > >> DRILL-5659 - problem is identified, Parth is working on the
>> fix.
>> > > >>
>> > > >> *Not blockers*
>> > > >> DRILL-5660 - Vitalii
>> > > >> DRILL-5468 - Jinfeng
>> > > >>
>> > > >> *Possible blockers*
>> > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
>> > > >> NPE on CSV - Paul
>> > > >>
>> > > >> Kind regards
>> > > >> Arina
>> > > >>
>> > > >>
>> > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <prog...@mapr.com
>> >
>> > wrote:
>> > > >>
>> > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle
>> for us
>> > to
>> > > >>> track down the incompatibility the issue internally, but
>> production
>> > > users
>> > > >>> probably don’t attempt to use one build of Drill with
>> metadata files
>> > > >>> created from another. We can worry about the issue later only
>> if a
&

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Latest update:

We wait for the following Jiras before final cut off.
1. DRILL-5601  - Paul
2. DRILL-5420 - Kunal
3. DRILL-5634 - Charles

Kind regards
Arina

On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kkha...@mapr.com> wrote:

> Parth's already +1'd the process with a minor update. So, we're good to go
> on DRILL-5420. I'll change the label on that to ready-to-commit.
>
> <http://www.mapr.com/>
>
> 
> From: Parth Chandra <par...@apache.org>
> Sent: Thursday, June 29, 2017 9:44:39 AM
> To: dev@drill.apache.org
> Subject: Re: [DISCUSS] Drill 1.11.0 release
>
> I'm Ok with the pcap format plugin if Pauls comments are also addressed. If
> the changes are done before your cutoff deadline,we can merge it in.
>
> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
>
> > Hi all,
> >
> > here is the list of Jira we are waiting before making final cut off:
> > 1. DRILL-5601  - Paul
> > 2. DRILL-3640  - Kunal
> > 3. DRILL-5420 - Kunal
> >
> > Please let me know if there are any others Jiras we should wait for.
> >
> > @Charles,
> > you have mentioned that you want to include some UDFs in Drill 1.11, do
> you
> > have any ETA for the PR?
> >
> > @Paul / Parth
> > I see there is active CR for pcap format PR, do we want to include it in
> > this release as well?
> >
> > Kind regards
> > Arina
> >
> >
> > On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> > > Parth, thanks a lot the instructions.
> > >
> > >
> > >
> > > On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <par...@apache.org>
> > wrote:
> > >
> > >> FWIW, I put release instructions culled from gists written by previous
> > >> release managers into an updated gist here [1]
> > >>
> > >> HTH
> > >>
> > >> Parth
> > >>
> > >> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> > >> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
> > >>
> > >>
> > >>
> > >> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <prog...@mapr.com>
> wrote:
> > >>
> > >> > Hi Arina,
> > >> >
> > >> > I have two projects in progress: one is a candidate for 1.11, the
> > other
> > >> is
> > >> > not.
> > >> >
> > >> > The “managed external sort” is the good candidate for 1.11: The PR
> for
> > >> > DRILL-5601 is being reviewed. Getting that in will allow us to
> enable
> > >> the
> > >> > “new” external sort by default.
> > >> >
> > >> > The other project is to address Drill’s memory fragmentation issue
> > with
> > >> > vectors over 16 MB in size. Two PRs have been open for a while. They
> > are
> > >> > the first of several. Each takes a while to review. So, the project
> > does
> > >> > not fit 1.11. The work can go into the next release instead.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > - Paul
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Parth, thanks a lot the instructions.



On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra  wrote:

> FWIW, I put release instructions culled from gists written by previous
> release managers into an updated gist here [1]
>
> HTH
>
> Parth
>
> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> 
>
>
>
> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers  wrote:
>
> > Hi Arina,
> >
> > I have two projects in progress: one is a candidate for 1.11, the other
> is
> > not.
> >
> > The “managed external sort” is the good candidate for 1.11: The PR for
> > DRILL-5601 is being reviewed. Getting that in will allow us to enable the
> > “new” external sort by default.
> >
> > The other project is to address Drill’s memory fragmentation issue with
> > vectors over 16 MB in size. Two PRs have been open for a while. They are
> > the first of several. Each takes a while to review. So, the project does
> > not fit 1.11. The work can go into the next release instead.
> >
> > Thanks,
> >
> > - Paul
> >
> >
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Hi all,

here is the list of Jira we are waiting before making final cut off:
1. DRILL-5601  - Paul
2. DRILL-3640  - Kunal
3. DRILL-5420 - Kunal

Please let me know if there are any others Jiras we should wait for.

@Charles,
you have mentioned that you want to include some UDFs in Drill 1.11, do you
have any ETA for the PR?

@Paul / Parth
I see there is active CR for pcap format PR, do we want to include it in
this release as well?

Kind regards
Arina


On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Parth, thanks a lot the instructions.
>
>
>
> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <par...@apache.org> wrote:
>
>> FWIW, I put release instructions culled from gists written by previous
>> release managers into an updated gist here [1]
>>
>> HTH
>>
>> Parth
>>
>> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
>> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
>>
>>
>>
>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <prog...@mapr.com> wrote:
>>
>> > Hi Arina,
>> >
>> > I have two projects in progress: one is a candidate for 1.11, the other
>> is
>> > not.
>> >
>> > The “managed external sort” is the good candidate for 1.11: The PR for
>> > DRILL-5601 is being reviewed. Getting that in will allow us to enable
>> the
>> > “new” external sort by default.
>> >
>> > The other project is to address Drill’s memory fragmentation issue with
>> > vectors over 16 MB in size. Two PRs have been open for a while. They are
>> > the first of several. Each takes a while to review. So, the project does
>> > not fit 1.11. The work can go into the next release instead.
>> >
>> > Thanks,
>> >
>> > - Paul
>> >
>> >
>> >
>> >
>>
>
>

Re: Webui and absolute links

[Arina Yelchiyeva]

Hi Jorg,

can you please first create Drill Jira and describe the problem there and
then link it to your PR?

Kind regards
Arina

On Mon, Aug 7, 2017 at 4:28 AM, Jörg Schad  wrote:

> Maybe code can express my intentions more clearly:
> https://github.com/apache/drill/pull/896
>
> On Sun, Aug 6, 2017 at 5:36 PM, Jörg Schad  wrote:
>
> > Looking at it, it seems to be a separate issue to me.
> >
> > That issue is about external references, I care about the internal links.
> > These currently use absolute addressing at many places, which leads to
> the
> > problem in the DC/OS UI that css files etc cannot be loaded:
> >
> >
> > On Sun, Aug 6, 2017 at 5:28 PM, Pritesh Maker  wrote:
> >
> >> Hi
> >>
> >> Could this be a duplicate of this JIRA - DRILL-5699: Drill Web UI Page
> >> Source Has Links To External Sites
> >>
> >> It’s in progress with this PR – https://github.com/apache/
> drill/pull/891
> >>
> >> Pritesh
> >>
> >> On 8/6/17, 4:28 PM, "Jörg Schad"  wrote:
> >>
> >> Hi,
> >> I am just creating a drill package for DC/OS / Apache Mesos.
> >>
> >> The one problem I am facing is the UI integration. Unfortunately the
> >> drill
> >> UI uses static links (
> >> e.g.,
> >> 
> >> 
> >> ) .
> >> This does not work nicely with the DC/OS UI, which would reference
> the
> >> drill UI via https:///service/drill/.
> >>
> >> Are there any objections against a PR making the all the links
> >> relative (
> >> e.g.,
> >> 
> >> 
> >> ).
> >>
> >>
> >> Thank you!
> >> Joerg
> >>
> >>
> >>
> >
>

Re: [ANNOUNCE] New Committer: Charles Givre

[Arina Yelchiyeva]

Congrats, Charles!

On Mon, Jun 12, 2017 at 7:47 PM, Parth Chandra  wrote:

> The Project Management Committee (PMC) for Apache Drill has invited Charles
> Givre to become a committer, and we are pleased to announce that he has
> accepted.
>
> Charles was instrumental in taking the HTTPD format plugin to completion
> and since then has remained active on the user and dev list. He also
> started a Drill UDFs list that he maintains here [1].
>
> Welcome Charles, and thank you for your contributions.  Keep up the good
> work !
>
> - Parth
> (on behalf of the Apache Drill PMC)
>
>
> [1] http://thedataist.com/drill-udfs/
>

Re: [ANNOUNCE] New PMC Chair of Apache Drill

[Arina Yelchiyeva]

Congratulations, Aman! 

Kind regards
Arina

> On Jun 23, 2017, at 10:37 PM, Abhishek Girish  wrote:
> 
> Congratulations Aman!
> 
>> On Fri, Jun 23, 2017 at 12:06 PM, Parth Chandra  wrote:
>> 
>> Hello Drill developers,
>> 
>>I'm very pleased to announce that the Drill PMC has voted to elect Aman
>> Sinha
>> as the new PMC chair of Apache Drill. Please join me in congratulating
>> Aman!
>> 
>>Thanks,
>> 
>> Parth
>> 

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

Hi Charles,

You create Drill Jira for this purpose but please add doc-impact label so
when your changes will be committed, functions are documented.
Drill stores UDFs in org.apache.drill.exec.expr.fn.impl package, you may
add your UDFs there. Please make sure your code complies with Drill
standards. Please add java doc describing general purpose of each function.
Regarding the unit tests, please take a look at String functions as example
[1, 2].

[1]
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java

[2]
https://github.com/apache/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestStringFunctions.java

Kind regards
Arina


On Mon, Jun 26, 2017 at 6:10 PM, Charles Givre <cgi...@gmail.com> wrote:

> Hi Arina,
> I have some UDFs which I’d love to include in version 1.11 which are
> available (and documented) here:
> https://github.com/cgivre/drill-crypto-functions <
> https://github.com/cgivre/drill-crypto-functions>
> and
> https://github.com/cgivre/drill-network-functions <
> https://github.com/cgivre/drill-network-functions>
>
> I’m not quite sure how to write unit tests for Drill UDFs so if someone
> could send me an example, I’ll do that and submit.  The crypto functions
> (MD5, SHA1) seem to be a fairly common request and the networking functions
> are a nice compliment for the PCAP functionality that Ted submitted.
>
> Thanks,
> — C
>
> > On Jun 23, 2017, at 07:45, Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> wrote:
> >
> > If there are no objections, I'd like to volunteer.
> >
> > Kind regards
> > Arina
> >
> > On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <par...@apache.org>
> wrote:
> >
> >> Hello Drillers,
> >>
> >>  It's been three months since the last release and it is time to do the
> >> next one. Does anyone want to volunteer to be the release manager? I
> would
> >> encourage our new committers to try their hand at it :)
> >>
> >>  While we wait for volunteers, if there are any issues on which work is
> in
> >> progress, that you feel we *must* include in the release, please post in
> >> reply to this thread.
> >>
> >> Thanks
> >>
> >> Parth
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

[Arina Yelchiyeva]

If there are no objections, I'd like to volunteer.

Kind regards
Arina

On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra  wrote:

> Hello Drillers,
>
>   It's been three months since the last release and it is time to do the
> next one. Does anyone want to volunteer to be the release manager? I would
> encourage our new committers to try their hand at it :)
>
>   While we wait for volunteers, if there are any issues on which work is in
> progress, that you feel we *must* include in the release, please post in
> reply to this thread.
>
> Thanks
>
> Parth
>

Re: [ANNOUNCE] New Committer: Paul Rogers

[Arina Yelchiyeva]

Congratulations Paul! Well deserved!

On Fri, May 19, 2017 at 6:23 PM, Charles Givre  wrote:

> Congrats Paul!!
>
> On Fri, May 19, 2017 at 11:22 AM, Aman Sinha  wrote:
>
> > The Project Management Committee (PMC) for Apache Drill has invited Paul
> > Rogers to become a committer, and we are pleased to announce that he has
> > accepted.
> >
> > Paul has a long list of contributions that have touched many aspects of
> the
> > product.
> >
> > Welcome Paul, and thank you for your contributions.  Keep up the good
> work
> > !
> >
> > - Aman
> >
> > (on behalf of the Apache Drill PMC)
> >
>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

I want to include DRILL-5337 (OpenTSDB plugin), I'll try to finish code
review during this week.

Kind regards
Arina

On Mon, Oct 9, 2017 at 4:08 PM, Arina Ielchiieva  wrote:

> Hi Drillers,
>
> It's been several months since the last release and it is time to do the
> next one. I am volunteering to be the release manager.
>
> If there are any issues on which work is in progress, that you feel we
> *must* include in the release, please post in reply to this thread. Based
> on your input we'll define release cut off date.
>
> Kind regards
> Arina
>

Re: Checkstyle Unused Imports

[Arina Yelchiyeva]

Also I might want to add check style for Apache header (which should be in
a form of comment, not Javadoc), agreed code style (like indents etc) and
enforced java doc for methods. At least the last two are enforced in
Calcite.
I used to point to all that stuff during code reviews but if all that would
be enforced, it would be much easier ...

Kind regards
Arina

On Sat, Sep 9, 2017 at 6:46 AM, Paul Rogers  wrote:

> Hi Vlad,
>
> Java has a wide variety of warnings available; each project decides which
> to ignore, which are warnings and which are errors. It may be that Eclipse,
> by default, has resource warnings turned on. The quick & dirty solution is
> simply to turn off warnings for AutoCloseables and missing @Overrides. This
> is, as they say, “crude but effective."
>
> It seems that the Drill community stand on imports is not to change them.
> Eclipse has an “organize imports” feature. I have to be careful when
> removing unused imports not to invoke this feature as it changes import
> order and often cause reviews to complain about unnecessary code changes.
>
> Would be good if we could 1) agree on a standard and 2) make sure that
> both Eclipse and IntelliJ can automatically organize imports to follow the
> standard. But, I personally don’t worry about imports because Eclipse takes
> care of it for me.
>
> For me, the bigger concern is about code style. Operators are implemented
> as huge, complex, deeply nested methods with many local variables (such as
> flags) set one place and used elsewhere — all with no comments. Would seem
> like a good idea to adopt best practices and require human-digestible
> method sizes with good Javadoc comments. To my mind, that will contribute
> more to the project than import order.
>
> Oh, and the other item that needs addressing is a requirement to create
> true unit tests (not just system tests coded with JUnit.) Good unit test
> will increase our code quality immensely, and will simplify the task for
> code reviews. So, I’d want to push that ahead before worrying about imports.
>
> Just my two cents…
>
> Thanks,
>
> - Paul
>
> > On Sep 8, 2017, at 6:58 PM, Vlad Rozov  wrote:
> >
> > Paul, is AutoCloseable warning specific to Eclipse? I don't remember
> seeing the same warning in IntelliJ or during compilation.
> >
> > I know that some communities are significantly more strict regarding
> code style and enforce not only unused imports, but also order of imports
> and placement of static imports. What is the Drill community stand on those
> items?
> >
> > Thank you,
> >
> > Vlad
> >
> > On 9/8/17 18:04, Paul Rogers wrote:
> >> I clean up the imports as I find them, but it would be nice to do them
> all at once to avoid the constant drip-drip-drop of warnings.
> >>
> >> The key problem is the generated code: the templates can’t really tell
> which imports are used where. So, we’d need to exclude generated code
> directories from the check style rules.
> >>
> >> Drill also has thousands of omitted “@Override” annotations and heavy
> abuse of AutoCloseable (which triggers warnings when used outside of
> try-with-resources).
> >>
> >> At present, Eclipse complains about 17,883 warnings in Drill code.
> >>
> >> - Paul
> >>
> >>> On Sep 8, 2017, at 4:43 PM, Timothy Farkas  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I've noticed that a lot of files have unused imports, and I frequently
> accidentally leave unused imports behind when I do refactoring. So I'd like
> to enable checkstyle to check for unused imports.
> >>>
> >>> Thanks,
> >>> Tim
> >
>
>

Re: Disable Lilith SOCKET connection by default.

[Arina Yelchiyeva]

I don't think that Lilith is critical for running unit tests. Since it is
used to view the logs, I guess you can disable it by default but please
leave the comment indicating the reason of the change.

Kind regards
Arina

On Thu, Aug 31, 2017 at 2:06 PM, Vova Vysotskyi  wrote:

> Hi all,
>
> I am having troubles when running Drill unit tests on the node where
> Hiveserver2 service is running. In this case, run hangs in the middle. It
> is because Lilith and Hiveserver2 use the same port 1.
> I tried to change the Lilith port in the logback.xml, but we cannot change
> this port in Lilith UI (https://github.com/huxi/lilith/issues/10).
> I have no other Ideas, how to avoid this issue without disabling Lilith or
> stopping HS2.
>
> Do you mind if I disable Lilith SOCKET connection by default?
>
> --
> Kind regards,
> Volodymyr Vysotskyi
>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status:

DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
reviewer - Paul) - fix is expected by the EOD in new pull request.
DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
reviewer - Paul) - unit test failures are fixed. Unit test performance
degraded x3 times!

Kind regards

On Sun, Nov 26, 2017 at 6:15 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Current status:
>
> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
> reviewer - Paul) - could not cherry-pick the commits. Needs fix.
> DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
> reviewer - Paul) - there are unit test failures. Needs fix.
>
> Kind regards
>
> On Sat, Nov 25, 2017 at 11:53 PM, AnilKumar B <akumarb2...@gmail.com>
> wrote:
>
>> Hi Arina,
>>
>> Sorry for the delay. Just now we squashed Kafka storage plugin commits
>> into
>> one commit and pushed.
>>
>> Thanks & Regards,
>> B Anil Kumar.
>>
>> On Sat, Nov 25, 2017 at 5:56 AM, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com> wrote:
>>
>> > Current status:
>> >
>> > DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
>> code
>> > reviewer - Paul) - needs to squash the commits.
>> > DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
>> > reviewer - Paul) - needs to address some code review comments.
>> >
>> > Kind regards
>> > Arina
>> >
>> > On Wed, Nov 15, 2017 at 2:38 PM, Arina Yelchiyeva <
>> > arina.yelchiy...@gmail.com> wrote:
>> >
>> > > Current status, we are close to the code freeze which will happen not
>> > > later then the end of the next week.
>> > >
>> > > Blocker:
>> > > DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R.,
>> code
>> > > reviewer - Arina) - in progress.
>> > >
>> > > Targeted for 1.12 release:
>> > > DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
>> code
>> > > reviewer - Paul) - needs next round of code review.
>> > > DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
>> > > mechanism (developer - Sorabh, code reviewer - Parth & Laurent) -
>> waiting
>> > > for Parth code review.
>> > > DRILL-5771: Fix serDe errors for format plugins (developer - Arina,
>> code
>> > > reviewer - Tim) - code review is done, waiting for the merge.
>> > >
>> > > Kind regards
>> > >
>> > > On Fri, Nov 10, 2017 at 9:32 AM, Chunhui Shi <c...@mapr.com> wrote:
>> > >
>> > >> Hi Arina,
>> > >>
>> > >>
>> > >> Could we consider to include DRILL-5089 in 1.12.0? It is about lazy
>> > >> loading schema for storage plugins. Could you or Paul take a look at
>> the
>> > >> pull request for this JIRA https://github.com/apache/drill/pull/1032?
>> I
>> > >> think both of you are familiar with this part.
>> > >>
>> > >>
>> > >> Thanks,
>> > >>
>> > >>
>> > >> Chunhui
>> > >>
>> > >> 
>> > >> From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
>> > >> Sent: Thursday, November 9, 2017 8:11:35 AM
>> > >> To: dev@drill.apache.org
>> > >> Subject: Re: [DISCUSS] Drill 1.12.0 release
>> > >>
>> > >> Yes, they are already in master.
>> > >>
>> > >> On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre <cgi...@gmail.com>
>> wrote:
>> > >>
>> > >> > We’re including the Networking functions in this release right?
>> > >> >
>> > >> > > On Nov 9, 2017, at 11:04, Arina Yelchiyeva <
>> > >> arina.yelchiy...@gmail.com>
>> > >> > wrote:
>> > >> > >
>> > >> > > If changes will be done before cut off date, targeting mid
>> November
>> > >> that
>> > >> > it
>> > >> > > will be possible to include this Jira.
>> > >> > >
>> > >> > > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com>
>> > >> wrote:
>> > >> > >
>> > >> > >> Hi Arina,
>> > >

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status:

DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
reviewer - Paul) - could not cherry-pick the commits. Needs fix.
DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
reviewer - Paul) - there are unit test failures. Needs fix.

Kind regards

On Sat, Nov 25, 2017 at 11:53 PM, AnilKumar B <akumarb2...@gmail.com> wrote:

> Hi Arina,
>
> Sorry for the delay. Just now we squashed Kafka storage plugin commits into
> one commit and pushed.
>
> Thanks & Regards,
> B Anil Kumar.
>
> On Sat, Nov 25, 2017 at 5:56 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
>
> > Current status:
> >
> > DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
> > reviewer - Paul) - needs to squash the commits.
> > DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
> > reviewer - Paul) - needs to address some code review comments.
> >
> > Kind regards
> > Arina
> >
> > On Wed, Nov 15, 2017 at 2:38 PM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> > > Current status, we are close to the code freeze which will happen not
> > > later then the end of the next week.
> > >
> > > Blocker:
> > > DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R.,
> code
> > > reviewer - Arina) - in progress.
> > >
> > > Targeted for 1.12 release:
> > > DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
> code
> > > reviewer - Paul) - needs next round of code review.
> > > DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
> > > mechanism (developer - Sorabh, code reviewer - Parth & Laurent) -
> waiting
> > > for Parth code review.
> > > DRILL-5771: Fix serDe errors for format plugins (developer - Arina,
> code
> > > reviewer - Tim) - code review is done, waiting for the merge.
> > >
> > > Kind regards
> > >
> > > On Fri, Nov 10, 2017 at 9:32 AM, Chunhui Shi <c...@mapr.com> wrote:
> > >
> > >> Hi Arina,
> > >>
> > >>
> > >> Could we consider to include DRILL-5089 in 1.12.0? It is about lazy
> > >> loading schema for storage plugins. Could you or Paul take a look at
> the
> > >> pull request for this JIRA https://github.com/apache/drill/pull/1032?
> I
> > >> think both of you are familiar with this part.
> > >>
> > >>
> > >> Thanks,
> > >>
> > >>
> > >> Chunhui
> > >>
> > >> 
> > >> From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> > >> Sent: Thursday, November 9, 2017 8:11:35 AM
> > >> To: dev@drill.apache.org
> > >> Subject: Re: [DISCUSS] Drill 1.12.0 release
> > >>
> > >> Yes, they are already in master.
> > >>
> > >> On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre <cgi...@gmail.com>
> wrote:
> > >>
> > >> > We’re including the Networking functions in this release right?
> > >> >
> > >> > > On Nov 9, 2017, at 11:04, Arina Yelchiyeva <
> > >> arina.yelchiy...@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > If changes will be done before cut off date, targeting mid
> November
> > >> that
> > >> > it
> > >> > > will be possible to include this Jira.
> > >> > >
> > >> > > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > >> Hi Arina,
> > >> > >> Can we include DRILL-4091 Support for additional GIS operations
> in
> > >> > version
> > >> > >> 1.12?  In general the code looked pretty good.  There was a unit
> > test
> > >> > >> missing which the developer submitted and some minor formatting
> > >> issues
> > >> > >> which I’m still waiting on.
> > >> > >> Thanks,
> > >> > >> —C
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>> On Nov 9, 2017, at 10:58, Arina Yelchiyeva <
> > >> arina.yelchiy...@gmail.com
> > >> > >
> > >> > >> wrote:
> > >> > >>>
> > >> > >>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status:

DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
reviewer - Paul) - needs to squash the commits.
DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
reviewer - Paul) - needs to address some code review comments.

Kind regards
Arina

On Wed, Nov 15, 2017 at 2:38 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Current status, we are close to the code freeze which will happen not
> later then the end of the next week.
>
> Blocker:
> DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., code
> reviewer - Arina) - in progress.
>
> Targeted for 1.12 release:
> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
> reviewer - Paul) - needs next round of code review.
> DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
> mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - waiting
> for Parth code review.
> DRILL-5771: Fix serDe errors for format plugins (developer - Arina, code
> reviewer - Tim) - code review is done, waiting for the merge.
>
> Kind regards
>
> On Fri, Nov 10, 2017 at 9:32 AM, Chunhui Shi <c...@mapr.com> wrote:
>
>> Hi Arina,
>>
>>
>> Could we consider to include DRILL-5089 in 1.12.0? It is about lazy
>> loading schema for storage plugins. Could you or Paul take a look at the
>> pull request for this JIRA https://github.com/apache/drill/pull/1032? I
>> think both of you are familiar with this part.
>>
>>
>> Thanks,
>>
>>
>> Chunhui
>>
>> 
>> From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
>> Sent: Thursday, November 9, 2017 8:11:35 AM
>> To: dev@drill.apache.org
>> Subject: Re: [DISCUSS] Drill 1.12.0 release
>>
>> Yes, they are already in master.
>>
>> On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre <cgi...@gmail.com> wrote:
>>
>> > We’re including the Networking functions in this release right?
>> >
>> > > On Nov 9, 2017, at 11:04, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com>
>> > wrote:
>> > >
>> > > If changes will be done before cut off date, targeting mid November
>> that
>> > it
>> > > will be possible to include this Jira.
>> > >
>> > > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com>
>> wrote:
>> > >
>> > >> Hi Arina,
>> > >> Can we include DRILL-4091 Support for additional GIS operations in
>> > version
>> > >> 1.12?  In general the code looked pretty good.  There was a unit test
>> > >> missing which the developer submitted and some minor formatting
>> issues
>> > >> which I’m still waiting on.
>> > >> Thanks,
>> > >> —C
>> > >>
>> > >>
>> > >>
>> > >>> On Nov 9, 2017, at 10:58, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com
>> > >
>> > >> wrote:
>> > >>>
>> > >>> Current status:
>> > >>>
>> > >>> Blocker:
>> > >>> DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R.,
>> > code
>> > >>> reviewer - ?) - in progress.
>> > >>>
>> > >>> Targeted for 1.12 release:
>> > >>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code
>> > >> reviewer -
>> > >>> Arina) - code review in final stage.
>> > >>> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
>> > code
>> > >>> reviewer - Paul) - in review.
>> > >>> DRILL-5943: Avoid the strong check introduced by DRILL-5582 for
>> PLAIN
>> > >>> mechanism (developer - Sorabh, code reviewer - Parth & Laurent) -
>> > waiting
>> > >>> for the code review.
>> > >>> DRILL-5771: Fix serDe errors for format plugins (developer - Arina,
>> > code
>> > >>> reviewer - Tim) - waiting for the code review.
>> > >>>
>> > >>> Kind regards
>> > >>> Arina
>> > >>>
>> > >>> On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva <
>> > >>> arina.yelchiy...@gmail.com> wrote:
>> > >>>
>> > >>>> Current status:
>> > >>>>
>> > >>>> Targeted for 1.12 release:
>> > >&

Re: drill vs hive vs native hbase count

[Arina Yelchiyeva]

Hi Dor,

I guess you need to provide more information. Environment are using (how
many drillbits, where data is located (fs, dfs))?
Data format you are querying? How many columns in file?

Kind regards
Arina

On Wed, Dec 13, 2017 at 2:06 PM, Dor Ben Dov  wrote:

> Anyone ?
> This is major integrity issue.
>
>
> Regards,
> Dor
>
> -Original Message-
> From: Dor Ben Dov
> Sent: יום ג 12 דצמבר 2017 13:28
> To: dev@drill.apache.org
> Subject: drill vs hive vs native hbase count
>
> Hi Everyone,
>
> I tried the same query of 'select count(*) from table x' from hive, drill
> gui (no hive) and natively using hbase count.
>
> The hive and hbase native got the same result, but in Drill - I got a
> different (higher) result.
>
> What might be or causing this gap ?
>
> Regards,
> Dor
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

Re: drill vs hive vs native hbase count

[Arina Yelchiyeva]

Hi Dor,

I guess we'll need more information.
If data is in Hbase and you query data from Drill which plugin are you
using (Hive / Hbase / MaprDB)?
What count result is expected? What result Drill returns?
Also as as Aman noted explain plan of query also would help.


Kind regards
Arina

On Thu, Dec 14, 2017 at 9:22 AM, Dor Ben Dov <dor.ben-...@amdocs.com> wrote:

> Hi Arina,
>
> I am using 3 driibits on my 3 instances of cloudera cluster.
> The data is located in Hbase and of course at the end as a file. How many
> columns - 10.
> The data is regular data but also jsons.
> Does this helps ?
>
> Regards,
> Dor
>
> -Original Message-
> From: Arina Yelchiyeva [mailto:arina.yelchiy...@gmail.com]
> Sent: יום ד 13 דצמבר 2017 19:21
> To: dev@drill.apache.org
> Subject: Re: drill vs hive vs native hbase count
>
> Hi Dor,
>
> I guess you need to provide more information. Environment are using (how
> many drillbits, where data is located (fs, dfs))?
> Data format you are querying? How many columns in file?
>
> Kind regards
> Arina
>
> On Wed, Dec 13, 2017 at 2:06 PM, Dor Ben Dov <dor.ben-...@amdocs.com>
> wrote:
>
> > Anyone ?
> > This is major integrity issue.
> >
> >
> > Regards,
> > Dor
> >
> > -Original Message-
> > From: Dor Ben Dov
> > Sent: יום ג 12 דצמבר 2017 13:28
> > To: dev@drill.apache.org
> > Subject: drill vs hive vs native hbase count
> >
> > Hi Everyone,
> >
> > I tried the same query of 'select count(*) from table x' from hive,
> > drill gui (no hive) and natively using hbase count.
> >
> > The hive and hbase native got the same result, but in Drill - I got a
> > different (higher) result.
> >
> > What might be or causing this gap ?
> >
> > Regards,
> > Dor
> >
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement,
> >
> > you may review at https://www.amdocs.com/about/email-disclaimer <
> > https://www.amdocs.com/about/email-disclaimer>
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement,
> >
> > you may review at https://www.amdocs.com/about/email-disclaimer <
> > https://www.amdocs.com/about/email-disclaimer>
> >
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi

[Arina Yelchiyeva]

Congratulations!

Kind regards
Arina

> On Dec 13, 2017, at 10:20 PM, Jinfeng Ni  wrote:
> 
> Congratulations and welcome, Boaz!
> 
> 
> Jinfeng
> 
> 
>> On Wed, Dec 13, 2017 at 11:17 AM, Robert Hou  wrote:
>> 
>> Congratulations, Boaz!
>> 
>> 
>> --Robert
>> 
>> 
>> From: Paul Rogers 
>> Sent: Wednesday, December 13, 2017 11:02 AM
>> To: dev@drill.apache.org
>> Subject: Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi
>> 
>> Congrats! Well deserved.
>> 
>> - Paul
>> 
>>> On Dec 13, 2017, at 11:00 AM, Timothy Farkas  wrote:
>>> 
>>> Congrats!
>>> 
>>> 
>>> From: Kunal Khatua 
>>> Sent: Wednesday, December 13, 2017 10:47:14 AM
>>> To: dev@drill.apache.org
>>> Subject: RE: [ANNOUNCE] New Committer: Boaz Ben-Zvi
>>> 
>>> Congratulations, Boaz!!
>>> 
>>> -Original Message-
>>> From: Abhishek Girish [mailto:agir...@apache.org]
>>> Sent: Wednesday, December 13, 2017 10:25 AM
>>> To: dev@drill.apache.org
>>> Subject: Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi
>>> 
>>> Congratulations Boaz!
 On Wed, Dec 13, 2017 at 10:23 AM Aman Sinha  wrote:
 
 The Project Management Committee (PMC) for Apache Drill has invited
 Boaz Ben-Zvi  to become a committer, and we are pleased to announce
 that he has accepted.
 
 Boaz has been an active contributor to Drill for more than a year.
 He designed and implemented the Hash Aggregate spilling and is leading
 the efforts for Hash Join spilling.
 
 Welcome Boaz, and thank you for your contributions.  Keep up the good
 work !
 
 - Aman
 (on behalf of the Apache Drill PMC)
 
>> 
>> 

Re: JSON reader enhancement

[Arina Yelchiyeva]

In general sounds good.
If user will apply kvgen / flatten over such 2-D array columns read as
string, he will be able to normalize data in the format he wants? Right? Or
we need to come up with new function?

Kind regards
Arina

On Fri, Nov 17, 2017 at 10:39 PM, Paul Rogers  wrote:

> Hi All,
>
> I’d like to propose a minor enhancement to the JSON reader to better
> handle non-relational JSON structures. (See DRILL-5974 [1].)
>
> As background, Drill handles simple tuples:
>
> {a: 10, b: “fred”}
>
> Drill also handles arrays:
>
> {name: “fred”, hobbies: [“bowling”, “golf”]}
>
> Drill even handles arrays of tuples:
>
> {name: “fred”, orders: [
>   {id: 1001, amount: 12.34},
>   {id: 1002, amount: 56.78}]}
>
> The above are termed "relational" because there is a straightforward
> mapping to/from tables into the above JSON structures.
>
> Things get interesting with non-relational types, such as 2-D arrays:
>
> {id: 4, shape: “square”, points: [[0, 0], [0, 5], [5, 0], [5, 5]]}
>
> Drill has two solutions:
>
> * Turn on the experimental list and union support.
> * Enable all-text mode to read all fields as JSON text.
>
> Here, I’d like to propose a middle ground:
>
> * Read fields with relational types into vectors.
> * Read non-relational fields using text mode.
>
> Thus, the first three examples would all result in the JSON data parsed
> into Drill vectors. But, the fourth, non-relational example would produce a
> row that looks like this:
>
> id, shape, points
> 4, “shape”, “[[0, 0], [0, 5], [5, 0], [5, 5]]”
>
> Although Drill can’t parse the 2-D array, Drill will pass the array along
> to the client, which can use its favorite JSON parser to parse the array
> and do something useful (like draw the square in this case.)
>
> In particular, the proposal:
>
> * Apply this change only to the revised “batch size aware” JSON reader.
> * Use the above parsing model by default.
> * Use the experimental list-and-union support if the existing
> `exec.enable_union_type` system/session option is set.
>
> Existing queries should “just work.” In fact, now JSON with non-relational
> types will work “out-of-the-box” without all-text mode or the experimental
> types.
>
> Thoughts?
>
> - Paul
>
> [1] https://issues.apache.org/jira/browse/DRILL-5974
>
>
>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status, we are close to the code freeze which will happen not later
then the end of the next week.

Blocker:
DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., code
reviewer - Arina) - in progress.

Targeted for 1.12 release:
DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
reviewer - Paul) - needs next round of code review.
DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - waiting
for Parth code review.
DRILL-5771: Fix serDe errors for format plugins (developer - Arina, code
reviewer - Tim) - code review is done, waiting for the merge.

Kind regards

On Fri, Nov 10, 2017 at 9:32 AM, Chunhui Shi <c...@mapr.com> wrote:

> Hi Arina,
>
>
> Could we consider to include DRILL-5089 in 1.12.0? It is about lazy
> loading schema for storage plugins. Could you or Paul take a look at the
> pull request for this JIRA https://github.com/apache/drill/pull/1032? I
> think both of you are familiar with this part.
>
>
> Thanks,
>
>
> Chunhui
>
> 
> From: Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> Sent: Thursday, November 9, 2017 8:11:35 AM
> To: dev@drill.apache.org
> Subject: Re: [DISCUSS] Drill 1.12.0 release
>
> Yes, they are already in master.
>
> On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre <cgi...@gmail.com> wrote:
>
> > We’re including the Networking functions in this release right?
> >
> > > On Nov 9, 2017, at 11:04, Arina Yelchiyeva <arina.yelchiy...@gmail.com
> >
> > wrote:
> > >
> > > If changes will be done before cut off date, targeting mid November
> that
> > it
> > > will be possible to include this Jira.
> > >
> > > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com>
> wrote:
> > >
> > >> Hi Arina,
> > >> Can we include DRILL-4091 Support for additional GIS operations in
> > version
> > >> 1.12?  In general the code looked pretty good.  There was a unit test
> > >> missing which the developer submitted and some minor formatting issues
> > >> which I’m still waiting on.
> > >> Thanks,
> > >> —C
> > >>
> > >>
> > >>
> > >>> On Nov 9, 2017, at 10:58, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com
> > >
> > >> wrote:
> > >>>
> > >>> Current status:
> > >>>
> > >>> Blocker:
> > >>> DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R.,
> > code
> > >>> reviewer - ?) - in progress.
> > >>>
> > >>> Targeted for 1.12 release:
> > >>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code
> > >> reviewer -
> > >>> Arina) - code review in final stage.
> > >>> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
> > code
> > >>> reviewer - Paul) - in review.
> > >>> DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
> > >>> mechanism (developer - Sorabh, code reviewer - Parth & Laurent) -
> > waiting
> > >>> for the code review.
> > >>> DRILL-5771: Fix serDe errors for format plugins (developer - Arina,
> > code
> > >>> reviewer - Tim) - waiting for the code review.
> > >>>
> > >>> Kind regards
> > >>> Arina
> > >>>
> > >>> On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva <
> > >>> arina.yelchiy...@gmail.com> wrote:
> > >>>
> > >>>> Current status:
> > >>>>
> > >>>> Targeted for 1.12 release:
> > >>>> DRILL-5832: Migrate OperatorFixture to use SystemOptionManager
> rather
> > >> than
> > >>>> mock (developer - Paul, code reviewer - ?) - waiting for the code
> > review
> > >>>> DRILL-5842: Refactor and simplify the fragment, operator contexts
> for
> > >>>> testing (developer - Paul, code reviewer - ?) - waiting for the code
> > >>>> review
> > >>>> DRILL-5834: Adding network functions (developer - Charles, code
> > reviewer
> > >>>> - Arina) - waiting changes after code review
> > >>>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer -
> > >> Arina) - waiting
> > >>>> for the code review
> > >>>> DRILL-5772: Enab

Re: gitbox?

[Arina Yelchiyeva]

+1 but as we have communicated on Hangouts committer should be responsible
for:

1. making sure that there is only one commit (otherwise ask developer to
squash the commits, unless several commits are required) and commit message
is according to the Drill standards.
2. all unit tests pass.
3. all functional and advanced tests pass (
https://github.com/mapr/drill-test-framework).

Kind regards
Arina

On Tue, Oct 31, 2017 at 11:53 PM, Charles Givre  wrote:

> I’d vote +1 for moving to gitbox.
> — C
>
> > On Oct 31, 2017, at 13:54, Parth Chandra  wrote:
> >
> > Bumping this thread up.
> >
> > Vlad brought this up in the hangout today and it sounds like we would
> like
> > to move to Gitbox. Thanks Vlad for the patient explanations!
> >
> > Committers, let's use this thread to vote on the the suggestion.
> >
> > I'm +1 on moving to gitbox.
> >
> > Also, I can work with Vlad and Paul on updating the merge process
> document.
> >
> >
> >
> > On Wed, Aug 30, 2017 at 1:34 PM, Vlad Rozov  wrote:
> >
> >> Hi,
> >>
> >> As I am new to Drill, I don't know if migration from "Git WiP" (
> >> https://git-wip-us.apache.org) to "Github Dual Master" (
> >> https://gitbox.apache.org) was already discussed by the community, but
> >> from my Apache Apex experience I would recommend to consider migrating
> >> Drill ASF repos to the gitbox. Such move will give committers write
> access
> >> to the Drill repository on Github with all the perks that Github
> provides.
> >>
> >> Thank you,
> >>
> >> Vlad
> >>
>
>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status:

Blocker:
DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., code
reviewer - ?) - in progress.

Targeted for 1.12 release:
DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code reviewer -
Arina) - code review in final stage.
DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
reviewer - Paul) - in review.
DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - waiting
for the code review.
DRILL-5771: Fix serDe errors for format plugins (developer - Arina, code
reviewer - Tim) - waiting for the code review.

Kind regards
Arina

On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Current status:
>
> Targeted for 1.12 release:
> DRILL-5832: Migrate OperatorFixture to use SystemOptionManager rather than
> mock (developer - Paul, code reviewer - ?) - waiting for the code review
> DRILL-5842: Refactor and simplify the fragment, operator contexts for
> testing (developer - Paul, code reviewer - ?) - waiting for the code
> review
> DRILL-5834: Adding network functions (developer - Charles, code reviewer
> - Arina) - waiting changes after code review
> DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer - Arina) - 
> waiting
> for the code review
> DRILL-5772: Enable UTF-8 support in query string by default (developer -
> Arina, code reviewer - Paul) - finalizing approach
> DRILL-4779: Kafka storage plugin support (developer - Anil, code reviewer
> - ?) - finishing implementation
>
> Under question:
> DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
> reviewer - ?) - waiting for the status update from the developer
>
> Please free to suggest other items that are targeted for 1.12 release.
> There are many Jiras that have fix version marked as 1.12, it would be good
> if developers revisit them and update fix version to the actual one.
> Link to the dashboard - https://issues.apache.org/
> jira/secure/RapidBoard.jspa?rapidView=185=DRILL=detail
>
> Kind regards
> Arina
>
>
> On Wed, Oct 11, 2017 at 2:42 AM, Parth Chandra <par...@apache.org> wrote:
>
>> I'm waiting to merge the SSL  changes in. Waiting a couple of days more to
>> see if there are any more comments before I merge the changes in.
>>
>> On Mon, Oct 9, 2017 at 10:28 AM, Paul Rogers <prog...@mapr.com> wrote:
>>
>> > Hi Arina,
>> >
>> > In addition to my own PRs, there are several in the “active” queue that
>> we
>> > could get in if we can just push them over the line and clear the queue.
>> > The owners of the PRs should check if we are waiting on them to take
>> action.
>> >
>> > 977 DRILL-5849: Add freemarker lib to dependencyManagement to ensure
>> > prop…
>> > 976 DRILL-5797: Choose parquet reader from read columns
>> > 975 DRILL-5743: Handling column family and column scan for hbase
>> > 973 DRILL-5775: Select * query on a maprdb binary table fails
>> > 972 DRILL-5838: Fix MaprDB filter pushdown for the case of nested
>> > field (reg. of DRILL-4264)
>> > 950 Drill 5431: SSL Support
>> > 949 DRILL-5795: Parquet Filter push down at rowgroup level
>> > 936 DRILL-5772: Add unit tests to indicate how utf-8 support can be
>> > enabled / disabled in Drill
>> > 904 DRILL-5717: change some date time test cases with specific
>> > timezone or Local
>> > 892 DRILL-5645: negation of expression causes null pointer exception
>> > 889 DRILL-5691: enhance scalar sub queries checking for the
>> cartesian
>> > join
>> >
>> > (Items not on the list above have become “inactive” for a variety of
>> > reasons.)
>> >
>> > Thanks,
>> >
>> > - Paul
>> >
>> > > On Oct 9, 2017, at 9:57 AM, Paul Rogers <prog...@mapr.com> wrote:
>> > >
>> > > Hi Arina,
>> > >
>> > > I’d like to include the following that are needed to finish up the
>> > “managed” sort and spill-to-disk for hash agg:
>> > >
>> > > #928: DRILL-5716: Queue-driven memory allocation
>> > > #958, DRILL-5808: Reduce memory allocator strictness for "managed"
>> > operators
>> > > #960, DRILL-5815: Option to set query memory as percent of total
>> > >
>> > > The following is needed to resolve issues with HBase support in empty
>> > batches:
>> > >
>> > > #968, DRILL-5830: Resolve regressions to MapR DB from DRI

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Yes, they are already in master.

On Thu, Nov 9, 2017 at 6:05 PM, Charles Givre <cgi...@gmail.com> wrote:

> We’re including the Networking functions in this release right?
>
> > On Nov 9, 2017, at 11:04, Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> wrote:
> >
> > If changes will be done before cut off date, targeting mid November that
> it
> > will be possible to include this Jira.
> >
> > On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com> wrote:
> >
> >> Hi Arina,
> >> Can we include DRILL-4091 Support for additional GIS operations in
> version
> >> 1.12?  In general the code looked pretty good.  There was a unit test
> >> missing which the developer submitted and some minor formatting issues
> >> which I’m still waiting on.
> >> Thanks,
> >> —C
> >>
> >>
> >>
> >>> On Nov 9, 2017, at 10:58, Arina Yelchiyeva <arina.yelchiy...@gmail.com
> >
> >> wrote:
> >>>
> >>> Current status:
> >>>
> >>> Blocker:
> >>> DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R.,
> code
> >>> reviewer - ?) - in progress.
> >>>
> >>> Targeted for 1.12 release:
> >>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code
> >> reviewer -
> >>> Arina) - code review in final stage.
> >>> DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh,
> code
> >>> reviewer - Paul) - in review.
> >>> DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
> >>> mechanism (developer - Sorabh, code reviewer - Parth & Laurent) -
> waiting
> >>> for the code review.
> >>> DRILL-5771: Fix serDe errors for format plugins (developer - Arina,
> code
> >>> reviewer - Tim) - waiting for the code review.
> >>>
> >>> Kind regards
> >>> Arina
> >>>
> >>> On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva <
> >>> arina.yelchiy...@gmail.com> wrote:
> >>>
> >>>> Current status:
> >>>>
> >>>> Targeted for 1.12 release:
> >>>> DRILL-5832: Migrate OperatorFixture to use SystemOptionManager rather
> >> than
> >>>> mock (developer - Paul, code reviewer - ?) - waiting for the code
> review
> >>>> DRILL-5842: Refactor and simplify the fragment, operator contexts for
> >>>> testing (developer - Paul, code reviewer - ?) - waiting for the code
> >>>> review
> >>>> DRILL-5834: Adding network functions (developer - Charles, code
> reviewer
> >>>> - Arina) - waiting changes after code review
> >>>> DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer -
> >> Arina) - waiting
> >>>> for the code review
> >>>> DRILL-5772: Enable UTF-8 support in query string by default
> (developer -
> >>>> Arina, code reviewer - Paul) - finalizing approach
> >>>> DRILL-4779: Kafka storage plugin support (developer - Anil, code
> >> reviewer
> >>>> - ?) - finishing implementation
> >>>>
> >>>> Under question:
> >>>> DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
> >>>> reviewer - ?) - waiting for the status update from the developer
> >>>>
> >>>> Please free to suggest other items that are targeted for 1.12 release.
> >>>> There are many Jiras that have fix version marked as 1.12, it would be
> >> good
> >>>> if developers revisit them and update fix version to the actual one.
> >>>> Link to the dashboard - https://issues.apache.org/
> >>>> jira/secure/RapidBoard.jspa?rapidView=185=
> DRILL=detail
> >>>>
> >>>> Kind regards
> >>>> Arina
> >>>>
> >>>>
> >>>> On Wed, Oct 11, 2017 at 2:42 AM, Parth Chandra <par...@apache.org>
> >> wrote:
> >>>>
> >>>>> I'm waiting to merge the SSL  changes in. Waiting a couple of days
> >> more to
> >>>>> see if there are any more comments before I merge the changes in.
> >>>>>
> >>>>> On Mon, Oct 9, 2017 at 10:28 AM, Paul Rogers <prog...@mapr.com>
> wrote:
> >>>>>
> >>>>>> Hi Arina,
> >>>>>>
> >>>>>> In addition to my own P

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

If changes will be done before cut off date, targeting mid November that it
will be possible to include this Jira.

On Thu, Nov 9, 2017 at 6:03 PM, Charles Givre <cgi...@gmail.com> wrote:

> Hi Arina,
> Can we include DRILL-4091 Support for additional GIS operations in version
> 1.12?  In general the code looked pretty good.  There was a unit test
> missing which the developer submitted and some minor formatting issues
> which I’m still waiting on.
> Thanks,
> —C
>
>
>
> > On Nov 9, 2017, at 10:58, Arina Yelchiyeva <arina.yelchiy...@gmail.com>
> wrote:
> >
> > Current status:
> >
> > Blocker:
> > DRILL-5917: Ban org.json:json library in Drill (developer - Vlad R., code
> > reviewer - ?) - in progress.
> >
> > Targeted for 1.12 release:
> > DRILL-5337: OpenTSDB plugin (developer - Dmitriy & Vlad S., code
> reviewer -
> > Arina) - code review in final stage.
> > DRILL-4779: Kafka storage plugin support (developer - Anil & Kamesh, code
> > reviewer - Paul) - in review.
> > DRILL-5943: Avoid the strong check introduced by DRILL-5582 for PLAIN
> > mechanism (developer - Sorabh, code reviewer - Parth & Laurent) - waiting
> > for the code review.
> > DRILL-5771: Fix serDe errors for format plugins (developer - Arina, code
> > reviewer - Tim) - waiting for the code review.
> >
> > Kind regards
> > Arina
> >
> > On Fri, Oct 20, 2017 at 1:49 PM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> >> Current status:
> >>
> >> Targeted for 1.12 release:
> >> DRILL-5832: Migrate OperatorFixture to use SystemOptionManager rather
> than
> >> mock (developer - Paul, code reviewer - ?) - waiting for the code review
> >> DRILL-5842: Refactor and simplify the fragment, operator contexts for
> >> testing (developer - Paul, code reviewer - ?) - waiting for the code
> >> review
> >> DRILL-5834: Adding network functions (developer - Charles, code reviewer
> >> - Arina) - waiting changes after code review
> >> DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer -
> Arina) - waiting
> >> for the code review
> >> DRILL-5772: Enable UTF-8 support in query string by default (developer -
> >> Arina, code reviewer - Paul) - finalizing approach
> >> DRILL-4779: Kafka storage plugin support (developer - Anil, code
> reviewer
> >> - ?) - finishing implementation
> >>
> >> Under question:
> >> DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
> >> reviewer - ?) - waiting for the status update from the developer
> >>
> >> Please free to suggest other items that are targeted for 1.12 release.
> >> There are many Jiras that have fix version marked as 1.12, it would be
> good
> >> if developers revisit them and update fix version to the actual one.
> >> Link to the dashboard - https://issues.apache.org/
> >> jira/secure/RapidBoard.jspa?rapidView=185=DRILL=detail
> >>
> >> Kind regards
> >> Arina
> >>
> >>
> >> On Wed, Oct 11, 2017 at 2:42 AM, Parth Chandra <par...@apache.org>
> wrote:
> >>
> >>> I'm waiting to merge the SSL  changes in. Waiting a couple of days
> more to
> >>> see if there are any more comments before I merge the changes in.
> >>>
> >>> On Mon, Oct 9, 2017 at 10:28 AM, Paul Rogers <prog...@mapr.com> wrote:
> >>>
> >>>> Hi Arina,
> >>>>
> >>>> In addition to my own PRs, there are several in the “active” queue
> that
> >>> we
> >>>> could get in if we can just push them over the line and clear the
> queue.
> >>>> The owners of the PRs should check if we are waiting on them to take
> >>> action.
> >>>>
> >>>> 977 DRILL-5849: Add freemarker lib to dependencyManagement to
> ensure
> >>>> prop…
> >>>> 976 DRILL-5797: Choose parquet reader from read columns
> >>>> 975 DRILL-5743: Handling column family and column scan for hbase
> >>>> 973 DRILL-5775: Select * query on a maprdb binary table fails
> >>>> 972 DRILL-5838: Fix MaprDB filter pushdown for the case of nested
> >>>> field (reg. of DRILL-4264)
> >>>> 950 Drill 5431: SSL Support
> >>>> 949 DRILL-5795: Parquet Filter push down at rowgroup level
> >>>> 936 DRILL-5772: Add unit tests to indicate how utf-8 support can
> be
> >>>&g

Re: Drill SASL Forward Compatibility

[Arina Yelchiyeva]

Hi Sorabh,

regarding your question:

>
> We have to either back port above forward compatibility fix into 1.10 and
> 1.11 or just fix in current release and support forward compatibility post
> 1.12+.
> Arina/Sudheesh - Please suggest if the analysis and fix sounds good and
> what are the releases we should consider the fix for. Considering 1.12
> release is planned soon can we take the fix in 1.12 release ?
>

I think we add the fix (if needed) in 1.12 and support forward
compatibility post 1.12+. As far as I know Drill never claimed it is
backward compatible and explicitly discourages users to use different Drill
versions in a cluster as well as for client and server.

Kind regards
Arina

On Wed, Nov 1, 2017 at 5:48 PM, Laurent Goujon  wrote:

> My comments inline:
>
> On Tue, Oct 31, 2017 at 6:11 PM, Sorabh Hamirwasia 
> wrote:
>
> > - if using Kerberos, things are a bit different: even if a MITM
> intercepts
> >
> > the token, only the server can decode it properly and set up encryption
> so
> > that only client can use it (a proxy server would not be able to decode
> the
> > traffic). So what you need to ensure is that you actually use Kerberos as
> > the only authentication mechanism, and that the channel is encrypted (if
> > channel is not encryted, see above). This is things you should do by
> > configuring the client by not sending the password (no need to), only
> > authorize Kerberos authentication, and verify that encryption is enabled
> > (which is already done I believe).
> >
> >
> > [Sorabh] - You are correct about the Kerberos preventing MITM which is
> > what I mentioned in the last response. But this is guaranteed if client
> and
> > server reach to the point of SASL handshake in their communication, since
> > SASL handshake exchanges all the bits related to Kerberos protocol.
> Before
> > that point is reached there are still few messages which is exchanged
> > between DrillClient and Drillbit to detect whether server side needs
> > authentication or not and what are supported mechanisms. This is where
> the
> > security flaw can cause client to believe Authentication is successfully
> > completed (even when Drillbit/DrillClient are authenticating using
> Kerberos
> > with or without encryption). This is what patch for DRILL-5582 is trying
> to
> > address.
> >
> >
> You still haven't answered what is the issue/security risk for the client
> here: sure the client didn't authenticate with the server, but at the same
> time it didn't get access to the server either...
>
> Also, it doesn't take very long to modify the rogue server to fake a SASL
> authentication. So, now you are "authenticated", but still not to the right
> server...
>
>
>
> >
> > As for the other issue at hand (compatibility between 1.11 client and
> 1.10
> > server), I am not sure to understand the proposed fix: is the logic you
> > proposed to be added to 1.10 server? why not simply add the missing enum
> > value (and that's it! no more values after that!)?
> >
> >
> > [Sorabh] - Just adding the missing enum value in 1.10 will not help since
> > in future if any other enum value is introduced then the same issue will
> be
> > seen. Moreover 1.10 doesn't support Encryption so that enum value should
> > not be added in that release. Instead the fix is to treat the return
> value
> > of UNKNOWN_SASL_SERVER while retrieving messages from future client as
> > valid default value and take decision based on that.
> >
>
> But 1.10.1 could consider that a client supporting encryption also support
> authentication? The code is already here in 1.10 btw:
> https://github.com/apache/drill/blob/1.10.0/exec/java-
> exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java#L297
>
> Alternatively, you could just check that the field sasl_support is set and
> not check the value alltogether. I'm not convinced you need to do some
> extra logic around UNKNOWN_SASL_SERVER which would just keep people
> confused (although it doesn't seem something you need to apply to 1.11 or
> higher)
>
>
>
> >
> >
> > Thanks,
> > Sorabh
> >
> > 
> > From: Laurent Goujon 
> > Sent: Tuesday, October 31, 2017 5:42 PM
> > To: dev
> > Cc: Arina Lelchieva; sudhe...@apache.org
> > Subject: Re: Drill SASL Forward Compatibility
> >
> > Regarding DRILL-5582 patch which broke compatibility with 1.9 version
> > (which is less than a year old):
> > I'm still not clear what we are trying to prevent here: stealing user
> > credentials and/or data? connecting to a rogue server which could return
> > corrupted data to the user? The patch gives a false sense of security
> > because it prevents none of it:
> > - if using password-based authentication, client is sending it in clear
> in
> > the initial handshake so I guess it's already game over!
> > - You could configure a client to not sent password over wire by choosing
> > another authentication algorithm, BUT if there's 

Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Arina Yelchiyeva]

Sorry, incorrectly estimated end date, it should be at 3:00 PM UTC (7:00 AM
PST), December 2, 2017.
So far we don't have enough votes to accept the release candidate. Please
vote!

Kind regards
Arina


On Thu, Nov 30, 2017 at 9:36 PM, Kunal Khatua  wrote:

> Performed sanity testing:
> Set up Drill on a physical node and ran a couple of queries consisting of
> Joins, aggregations, etc against the TPCH dataset.
>
> +1 (non-binding)
>
>
> -Original Message-
> From: Aman Sinha [mailto:amansi...@apache.org]
> Sent: Thursday, November 30, 2017 8:52 AM
> To: dev@drill.apache.org
> Subject: Re: [VOTE] Release Apache Drill 1.12.0 - rc0
>
> - Downloaded the source tarball from [2] on my Linux VM
> - Built and ran all unit tests - took about 36 minutes.  Monitored the
> memory usage on the machine periodically, looked good.
> - Started Drill in embedded mode and manually ran a bunch of sanity
> queries with joins, aggregations, order-by and CTAS statements, against a
> TPCH dataset.
> - Examined the run-time query profiles of several queries with parallelism.
> - Checked the maven artifacts on [3].
>
> LGTM  +1 (binding)
>
> Thanks Arina for guiding the release and thanks to all contributors who
> worked on the release !
>
>
> On Wed, Nov 29, 2017 at 6:50 AM, Arina Ielchiieva 
> wrote:
>
> > Hi all,
> >
> > I'd like to propose the first release candidate (rc0) of Apache Drill,
> > version 1.12.0.
> >
> > The release candidate covers a total of 167 resolved JIRAs [1]. Thanks
> > to everyone who contributed to this release.
> >
> > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted at [3].
> >
> > This release candidate is based on commit
> > 54d3d201882ef5bc2e0f754fd10edfead9947b60 located at [4].
> >
> > The vote ends at 3:00 PM UTC (7:00 AM PT), December 1, 2017.
> >
> > [ ] +1
> > [ ] +0
> > [ ] -1
> >
> > Here's my vote: +1
> >
> >
> > [1]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org
> > _jira_secure_ReleaseNote.jspa-3F=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=-
> > cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU=
> > G7nd0QG8searR-Ynr_L0uwLMLVPPrZQM7-kLDX6CzLg=
> > projectId=12313820=12341087
> > [2]
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7
> > Earina_drill_releases_1.12.0_rc0_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=
> > -cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU
> > =Wip_bHewQWUXZfGZbZcWpv2BtSkeSHIuyxaDwBOnwXA=
> > [3]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache
> > .org_content_repositories_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=-cT6otg
> > 6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU=A2sU7W
> > RWS4GRHVQeFf3oKlT1iWB1bxkXtmrZcQHVqT8=
> > orgapachedrill-1043/
> > [4]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_arina-
> > 2Dielchiieva_drill_commits_drill-2D1.12.0=DwIBaQ=cskdkSMqhcnjZxdQV
> > pwTXg=-cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL
> > -NRZVU=W9VClczvLQyBXfLsATbXDCX8QxIRddSH8ap3kyU5YVY=
> >
> > Kind regards
> > Arina
> >
>

Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Arina Yelchiyeva]

I guess this sinks the release. I'll prepare new release candidate once
issue in DRILL-5702 [1] is fixed.

[1] https://issues.apache.org/jira/browse/DRILL-5702

Kind regards
Arina

On Sat, Dec 2, 2017 at 12:14 PM, Vova Vysotskyi  wrote:

> Downloaded built archive at [2], ran Drill in embedded mode, created and
> queried views, submitted few TPCH queries from UI, checked that query
> profiles are displayed correctly.
> Connected from SQuirrel using drill-jdbc-driver driver, ran few queries,
> also ran queries from a java application using drill-jdbc-driver, all works
> fine for me.
> Downloaded source archive at [2], ran unit tests, all tests are passed on
> my pc.
>
> +1 (non-binding)
>
> 2017-12-02 0:36 GMT+02:00 Abhishek Girish :
>
> > I'm seeing a unit test failure tracked by DRILL-6003, when building from
> > [4]. Can someone take a look?
> >
> > On Wed, Nov 29, 2017 at 6:50 AM, Arina Ielchiieva 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to propose the first release candidate (rc0) of Apache Drill,
> > > version 1.12.0.
> > >
> > > The release candidate covers a total of 167 resolved JIRAs [1]. Thanks
> to
> > > everyone who contributed to this release.
> > >
> > > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted
> > > at [3].
> > >
> > > This release candidate is based on commit
> > > 54d3d201882ef5bc2e0f754fd10edfead9947b60 located at [4].
> > >
> > > The vote ends at 3:00 PM UTC (7:00 AM PT), December 1, 2017.
> > >
> > > [ ] +1
> > > [ ] +0
> > > [ ] -1
> > >
> > > Here's my vote: +1
> > >
> > >
> > > [1]
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12313820=12341087
> > > [2] http://home.apache.org/~arina/drill/releases/1.12.0/rc0/
> > > [3] https://repository.apache.org/content/repositories/
> > > orgapachedrill-1043/
> > > [4] https://github.com/arina-ielchiieva/drill/commits/drill-1.12.0
> > >
> > > Kind regards
> > > Arina
> > >
> >
>
>
>
> --
> Kind regards,
> Volodymyr Vysotskyi
>

Re: [ANNOUNCE] New Committer: Vitalii Diravka

[Arina Yelchiyeva]

Congratulations!

Kind regards
Arina

On Mon, Dec 11, 2017 at 8:06 AM, Prasad Nagaraj Subramanya <
prasadn...@gmail.com> wrote:

> Congratulations Vitali!
>
> Thanks,
> Prasad
>
> On Sun, Dec 10, 2017 at 9:58 PM, Robert Hou  wrote:
>
> > Congratulations!
> >
> >
> > --Robert
> >
> > 
> > From: Paul Rogers 
> > Sent: Sunday, December 10, 2017 4:29 PM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
> >
> > Congrats! Well deserved.
> >
> > - Paul
> >
> > > On Dec 10, 2017, at 3:16 PM, AnilKumar B 
> wrote:
> > >
> > > Congratulations Vitalii
> > >
> > > Thanks & Regards,
> > > B Anil Kumar.
> > >
> > > On Sun, Dec 10, 2017 at 3:12 PM, rahul challapalli <
> > > challapallira...@gmail.com> wrote:
> > >
> > >> Congratulations Vitalii!
> > >>
> > >> On Sun, Dec 10, 2017 at 3:05 PM, Kunal Khatua 
> wrote:
> > >>
> > >>> Congratulations!!
> > >>>
> > >>> -Original Message-
> > >>> From: Aman Sinha [mailto:amansi...@apache.org]
> > >>> Sent: Sunday, December 10, 2017 11:06 AM
> > >>> To: dev@drill.apache.org
> > >>> Subject: [ANNOUNCE] New Committer: Vitalii Diravka
> > >>>
> > >>> The Project Management Committee (PMC) for Apache Drill has invited
> > >>> Vitalii Diravka  to become a committer, and we are pleased to
> announce
> > >> that
> > >>> he has accepted.
> > >>>
> > >>> Vitalii has been an active contributor to Drill over the last 1 1/2
> > >> years.
> > >>> His contributions have spanned areas such as: CASTing issues with
> > >>> Date/Timestamp, Parquet metadata and SQL enhancements, among others.
> > >>>
> > >>> Welcome Vitalii, and thank you for your contributions.  Keep up the
> > good
> > >>> work !
> > >>>
> > >>> - Aman
> > >>> (on behalf of the Apache Drill PMC)
> > >>>
> > >>
> >
> >
>

Re: [IMPORTANT] Gitbox enabled

[Arina Yelchiyeva]

Another alternative is to use ssh [1].

[1]
https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/


Kind regards
Arina

On Thu, May 10, 2018 at 2:27 AM Vlad Rozov  wrote:

> See [1] and [2]
>
> Thank you,
>
> Vlad
>
> [1]
>
> https://help.github.com/articles/providing-your-2fa-authentication-code/#when-youll-be-asked-for-a-personal-access-token-as-a-password
> [2]
>
> https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
>
>
> On 5/9/18 16:12, Boaz Ben-Zvi wrote:
> >   Note *committers* , in case you get the same error:
> >
> > After successfully enabling the needed Two Factor Authentication (2FA),
> my “git push” started failing, like:
> >
> > ~/drill > git push origin
> > Username for 'https://github.com': ben-zvi
> > Password for 'https://ben-...@github.com':
> > remote: Invalid username or password.
> > fatal: Authentication failed for 'https://github.com/Ben-Zvi/drill.git/'
> >
> > The solution: Need to enter a personal access token instead of the
> github password.
> > To generate a personal access token, go to
> https://github.com/settings/tokens
> > The token is a long hash code ; just copy and paste it as a password.
> >
> > Thanks,
> >
> > Boaz
> >
> >
> > On 5/3/18, 11:36 AM, "Parth Chandra"  wrote:
> >
> >  Note to all the *committers* -
> >
> >  Gitbox integration has been enabled. This means you can merge in a
> PR
> >  directly from Github. (i.e. the apache/drill repository on github
> is now
> >  the master repository, and  is writable. (It is no longer a mirror).
> >
> >  This also means that the original git-wip repository will not be
> available
> >  and pushing to this repository will not achieve anything useful.
> >
> >  [IMPORTANT] Please visit
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitbox.apache.org_setup_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU=eWZe_PpBWgOlhFxb0vKBmQ0-DhAbSxF_WoJ8rg8dT9U=fsXeKdxWoc0QL7vvlOPkm4D4aiyv_gLn0IM1oP8s7TM=
> to setup 2FA if
> >  you'd like to use GitHub as a remote g...@github.com:
> apache/drill.git
> >
> >  You can also use GitBox as a remote
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitbox.apache.org_repos_asf_drill.git=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU=eWZe_PpBWgOlhFxb0vKBmQ0-DhAbSxF_WoJ8rg8dT9U=DOiW8QP9zBVK4JKcS-aDmTsrOvQJu7syQgTWpCRi2zM=
> >
> >  Same thing for drill-site g...@github.com:apache/drill-site.git or
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitbox.apache.org_repos_asf_drill-2Dsite.git=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU=eWZe_PpBWgOlhFxb0vKBmQ0-DhAbSxF_WoJ8rg8dT9U=XKrIn1R2oXPsF_Y4OPadWjIeRQXAOv7BJgEi-qWY-Kg=
> >
> >
> >  [IMPORTANT] - The github UI currently enables the option to "Create
> a merge
> >  commit" . Please *do not* use this option. Click on the drop down
> and chose
> >  the "rebase and merge" or "squash and merge" option
> >
> >  @vrozov is the expert on this, so if you run into difficulties
> please
> >  include him in the communication. (Better still just post on the
> list).
> >
> >  Thanks
> >
> >  Parth
> >
> >
>
>

Re: [Notice] CheckStyle Changes :)

[Arina Yelchiyeva]

Nice :) Thanks, Tim!

On Fri, May 11, 2018 at 2:28 AM Timothy Farkas  wrote:

> Hi All,
>
> Unused imports have been recently disallowed, so Travis and Jenkins will
> fail if your PR includes unused imports. To avoid breaking the build,
> please rebase your PRs onto the latest master and run "mvn
> checkstyle:check" and fix any errors before your PRs are merged.
>
> Thanks,
> Tim
>
>
>

Re: [ANNOUNCE] New Committer: Timothy Farkas

[Arina Yelchiyeva]

Congrats, Tim! 

Kind regards,
Arina

> On May 25, 2018, at 9:59 PM, Kunal Khatua  wrote:
> 
> Congratulations, Timothy !
> 
> On 5/25/2018 11:58:31 AM, Aman Sinha  wrote:
> The Project Management Committee (PMC) for Apache Drill has invited Timothy
> Farkas to become a committer, and we are pleased to announce that he
> has accepted.
> 
> Tim has become an active contributor to Drill in less than a year. During
> this time he has contributed to addressing flaky unit tests, fixing memory
> leaks in certain operators, enhancing the system options framework to be
> more extensible and setting up the Travis CI tests. More recently, he
> worked on the memory sizing calculations for hash join.
> 
> Welcome Tim, and thank you for your contributions. Keep up the good work !
> 
> -Aman
> (on behalf of Drill PMC)

Re: [DISCUSS] case insensitive storage plugin and workspaces names

[Arina Yelchiyeva]

To make it clear we have three notions here: storage plugin name, workspace
(schema) and table name (dfs.root.`/tmp/t`).
My suggestion is the following:
Storage plugin names to be case insensitive (DFS vs dfs, INFORMATION_SCHEMA
vs information_schema).
Workspace  (schemas) names to be case insensitive (ROOT vs root, TMP vs
tmp). Even if user has two directories /TMP and /tmp, he can create two
workspaces but not both with tmp name. For example, tmp vs tmp_u.
Table names case sensitivity are treated per plugin. For example, system
plugins (information_schema, sys) table names (views, tables) should be
case insensitive. Actually, currently for sys plugin table names are case
insensitive, information_schema table names are case sensitive. That needs
to be synchronized. For file system plugins table names must be case
sensitive, since under table name we imply directory / file name and their
case sensitivity depends on file system.

Kind regards,
Arina

On Tue, Jun 12, 2018 at 6:13 PM Aman Sinha  wrote:

> Drill is dependent on the underlying file system's case sensitivity.  On
> HDFS one can create  'hadoop fs -mkdir /tmp/TPCH'  and /tmp/tpch which are
> separate directories.
> These could be set as workspace in Drill's storage plugin configuration and
> we would want the ability to query both.   If we change the current
> behavior, we would want
> some way, either using back-quotes `  or other way to support that.
>
> RDBMSs seem to have vendor-specific behavior...
> In MySQL [1] the database name and schema name are case-sensitive on Linux
> and case-insensitive on Windows.   Whereas in Postgres it converts the
> database name and schema name to lower-case by default but one can put
> double-quotes to make it case-sensitive [2].
>
> [1]
> https://dev.mysql.com/doc/refman/8.0/en/identifier-case-sensitivity.html
> [2]
> http://www.postgresqlforbeginners.com/2010/11/gotcha-case-sensitivity.html
>
>
>
> On Tue, Jun 12, 2018 at 5:01 AM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
>
> > Hi all,
> >
> > Currently Drill we treat storage plugin names and workspaces as
> > case-sensitive [1].
> > Names for storage plugins and workspaces are defined by the user. So we
> > allow to create plugin -> DFS and dfs, workspace -> tmp and TMP.
> > I have a suggestion to move to case insensitive approach and won't allow
> > creating two plugins / workspaces with the same name in different case at
> > least for the following reasons:
> > 1. usually rdbms schema and table names are case insensitive and many
> users
> > are used to this approach;
> > 2. in Drill we have INFORMATION_SCHEMA schema which is in upper case, sys
> > in lower case.
> > personally I find it's extremely inconvenient.
> >
> > Also we should consider making table names case insensitive for system
> > schemas (info, sys).
> >
> > Any thoughts?
> >
> > [1] https://drill.apache.org/docs/lexical-structure/
> >
> >
> > Kind regards,
> > Arina
> >
>

Re: [DISCUSS] case insensitive storage plugin and workspaces names

[Arina Yelchiyeva]

>From the Drill code workspaces are already case insensitive (though the
documentation states the opposite). Since there were no complaints from the
users so far, I believe there are not many (if any) who uses the same names
in different case.
Regarding those users that already have duplicating storage plugins names,
after the change Drill start up will fail with appropriate error message
and they would have to rename those storage plugins.

Kind regards,
Arina


On Tue, Jun 12, 2018 at 8:45 PM Abhishek Girish  wrote:

> Paul, I think this proposal was specific to storage plugin and workspace
> *names*. And not for the whole of Drill.
>
> I agree it makes sense to have these names case insensitive, to improve
> user experience. The only impact to current users I can think of is if
> someone created two storage plugins dfs and DFS. Or configured workspaces
> tmp and TMP. In this case, they'd need to rename those. One thing I'm not
> clear on is how we'll handle upgrades in these cases.
>
> On Tue, Jun 12, 2018 at 10:31 AM Paul Rogers 
> wrote:
>
> > Hi All,
> >
> > As it turns out, this topic has been discussed, in depth, previously.
> > Can't recall if it was on this list, or in a JIRA.
> >
> > We face a number of constraints:
> >
> > * As was noted, for some data sources, the data source itself has case
> > insensitive names. (Windows file systems, RDBMSs, etc.)
> > * In other cases, the data source itself has case sensitive names. (HDFS
> > file system, Linux file systems, JSON, etc.)
> > * SQL is defined to be case insensitive.
> > * We now have several years of user queries, in production, based on the
> > current semantics.
> >
> > Given all this, it is very likely that simply shifting to case-sensitive
> > will break existing applications.
> >
> > Perhaps a more subtle solution is to make the case-sensitivity a property
> > of the symbol that is carried through the query pipeline as another piece
> > of metadata.
> >
> > Thus, a workspace that corresponds to a DB schema would be labeled as
> case
> > insensitive. A workspace that corresponds to an HDFS directory would be
> > case sensitive. Names defined within Drill (as part of an AS clause),
> would
> > follow SQL rules and be case insensitive.
> >
> > I believe that, if we sit down and work out exactly what users would
> > expect, and what is required to handle both case sensitive and case
> > insensitive names, we'll end up with a solution not far from the above --
> > out of simple necessity.
> >
> > Thanks,
> > - Paul
> >
> >
> >
> > On Tuesday, June 12, 2018, 8:36:01 AM PDT, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> >  To make it clear we have three notions here: storage plugin name,
> > workspace
> > (schema) and table name (dfs.root.`/tmp/t`).
> > My suggestion is the following:
> > Storage plugin names to be case insensitive (DFS vs dfs,
> INFORMATION_SCHEMA
> > vs information_schema).
> > Workspace  (schemas) names to be case insensitive (ROOT vs root, TMP vs
> > tmp). Even if user has two directories /TMP and /tmp, he can create two
> > workspaces but not both with tmp name. For example, tmp vs tmp_u.
> > Table names case sensitivity are treated per plugin. For example, system
> > plugins (information_schema, sys) table names (views, tables) should be
> > case insensitive. Actually, currently for sys plugin table names are case
> > insensitive, information_schema table names are case sensitive. That
> needs
> > to be synchronized. For file system plugins table names must be case
> > sensitive, since under table name we imply directory / file name and
> their
> > case sensitivity depends on file system.
> >
> > Kind regards,
> > Arina
> >
> > On Tue, Jun 12, 2018 at 6:13 PM Aman Sinha  wrote:
> >
> > > Drill is dependent on the underlying file system's case sensitivity.
> On
> > > HDFS one can create  'hadoop fs -mkdir /tmp/TPCH'  and /tmp/tpch which
> > are
> > > separate directories.
> > > These could be set as workspace in Drill's storage plugin configuration
> > and
> > > we would want the ability to query both.  If we change the current
> > > behavior, we would want
> > > some way, either using back-quotes `  or other way to support that.
> > >
> > > RDBMSs seem to have vendor-specific behavior...
> > > In MySQL [1] the database name and schema name are case-sensitive on
> > Linux
> > > and case-insensitive on Windows.  Whereas in Postgres it converts the
> > &

Re: DRILL-6519 Tavis CI timing out

[Arina Yelchiyeva]

We are just finalizing the list of tests we are going to be marked as slow.
Once done, this would be an easy fix :)

Kind regards,
Arina

On Fri, Jun 29, 2018 at 7:41 PM Charles Givre  wrote:

> Hi Vitalii
> I saw the discussion but is there anything that I can do to prevent this
> from happening?
>
> Sent from my iPhone
>
> > On Jun 29, 2018, at 12:34, Vitalii Diravka 
> wrote:
> >
> > Hi Charles,
> >
> > Recently there was a discussion in dev mailing list regarding this.
> Please
> > take a look.
> > Also I have created Jira ticket [1].
> >
> > [1] https://issues.apache.org/jira/browse/DRILL-6559
> >
> > Kind regards
> > Vitalii
> >
> >
> >> On Fri, Jun 29, 2018 at 5:43 PM Charles Givre  wrote:
> >>
> >> Hi Arina,
> >> Could you or someone take a look at the CI.  I’ve submitted the changes,
> >> the CI seems to build successfully, but times out and thus fails the
> check.
> >> Thanks,
> >> — C
>

Re: [DISCUSSION] Travis build failures

[Arina Yelchiyeva]

Or we can be more radical here and exclude all tests from contib module?

On Fri, Jun 29, 2018 at 7:28 PM Vitalii Diravka 
wrote:

> Agree, TestTpchExplain and TestTpchPlanning are fast enough.
> TestTpchLimit0 has PlannerTest category, but it is slower than the above
> tests.
> Actually only TestTpchSingleMode and TestTpchLimit0 are candidates for
> excluding.
> I am not sure that it is enough for Travis (45 seconds on my machine). What
> about Mongo unit tests?
>
> I have created Jira for it [1].
>
> [1] https://issues.apache.org/jira/browse/DRILL-6559
>
> Kind regards
> Vitalii
>
>
> On Wed, Jun 27, 2018 at 10:29 PM Aman Sinha  wrote:
>
> > Sounds good but why exclude the planning tests ?  Only the TPC-H runtime
> > tests should be excluded I think.
> >
> > On Wed, Jun 27, 2018 at 12:16 PM Timothy Farkas 
> wrote:
> >
> > > +1
> > >
> > > On Wed, Jun 27, 2018 at 10:00 AM, Vitalii Diravka <
> > > vitalii.dira...@gmail.com
> > > > wrote:
> > >
> > > > This is a topic from last Hangout meeting.
> > > >
> > > > Sometimes Drill Travis Build fails because of job run expires.
> > > > Certainly, the right way is to accelerate Drill execution :)
> > > >
> > > > Nevertheless I believe we could consider excluding some more tests
> from
> > > > Travis Build.
> > > > We can add all TPCH tests (
> > > > TestTpchLimit0, TestTpchExplain, TestTpchPlanning, TestTpchExplain)
> to
> > > the
> > > > SlowTest category.
> > > > Actually it isn't urgent, but we can consider it for future, if this
> > > > happens more often.
> > > >
> > > > Kind regards
> > > > Vitalii
> > > >
> > >
> >
>

Re: [ANNOUNCE] New PMC member: Vitalii Diravka

[Arina Yelchiyeva]

Congratulations, Vitalii! Well deserved!

Kind regards,
Arina

On Tue, Jun 26, 2018 at 9:16 PM Bridget Bevens  wrote:

> Congratulations, Vitalii!
>
> On Tue, Jun 26, 2018 at 11:14 AM, Abhishek Girish 
> wrote:
>
> > Congratulations, Vitalii!
> >
> > On Tue, Jun 26, 2018 at 11:12 AM Aman Sinha 
> wrote:
> >
> > > I am pleased to announce that Drill PMC invited Vitalii Diravka to the
> > PMC
> > > and he has accepted the invitation.
> > >
> > > Congratulations Vitalii and thanks for your contributions !
> > >
> > > -Aman
> > > (on behalf of Drill PMC)
> > >
> >
>

Re: Excessive review comments

[Arina Yelchiyeva]

Ideally, there should be an option if you want to receive all notifications
from github or not. All these notifications don't really bother me. As
Parth mentioned this is the way to track dev activity.
Subscribing to the pull request maybe not always be the best solution,
since sometimes the comments in pull request is the point of interest and
gets you engaged into the discussion.

Regarding the code review comments, I think we should not point to the code
reviewers how they should write comments until they provide good code
review and raise good questions.
Especially taking into account limited number of volunteers to do the code
review...


Kind regards
Arina

On Thu, Oct 19, 2017 at 9:55 PM, Aman Sinha <amansi...@apache.org> wrote:

> Going back to one comment by Paul : “The only other solution is to give few
> review comments; not sure if we want to go that route...”
> IMO the individual code review comments should be concise such that the
> main idea is expressed.  This is more palatable for the original developer
> and he/she can act on it.  If there are several comments that are somewhat
> lengthier, it amounts to a design discussion.  These would be better
> combined and could either be (a) added to the JIRA or (b) added to the
> ‘conversation’ section of the PR to minimize notifications.
>
> -Aman
>
> On Thu, Oct 19, 2017 at 10:59 AM, Padma Penumarthy <ppenumar...@mapr.com>
> wrote:
>
> > To subscribe to PRs and JIRAs of interest, we need to know about them
> > first.
> > Is it possible to get notified when a new PR or JIRA is created and not
> for
> > further updates unless you subscribe to them ?
> >
> > Thanks
> > Padma
> >
> >
> > > On Oct 19, 2017, at 10:13 AM, Paul Rogers <prog...@mapr.com> wrote:
> > >
> > > So, rather than copying all messages to dev, simply ask interested
> folks
> > subscribe to the PRs of interest. (Or, subscribe to the corresponding
> JIRA
> > tickets, since JIRA echos all comments in another spray of e-mail…)
> > >
> > > This way, semi-active folks on the dev list can see substantive
> > discussions without being bombarded with day-to-day minutia.
> > >
> > > Thanks,
> > >
> > > - Paul
> > >
> > >> On Oct 19, 2017, at 10:07 AM, Parth Chandra <par...@apache.org>
> wrote:
> > >>
> > >> I generally keep track of dev activity only through the emails sent. I
> > >> don't mind getting duplicate emails either - it's not too hard to
> write
> > a
> > >> filter to take out the stuff you're not interested in.
> > >>
> > >> Interestingly, I've never noticed whether 'start a review' sends one
> > email
> > >> or many; mostly because gmail does a nice job of grouping the emails
> > >> together. I have seen a single email with many comments being sent out
> > so
> > >> the feature did work as advertised at one point.
> > >>
> > >> That said, if there is a way I can stay updated via email (repo
> > watching,
> > >> jira updates, etc.) then I am fine with turning the feature off.
> > >>
> > >>
> > >>
> > >> On Thu, Oct 19, 2017 at 9:51 AM, Timothy Farkas <tfar...@mapr.com>
> > wrote:
> > >>
> > >>> +1 for turning off the feature. If someone really needs to be emailed
> > with
> > >>> comment updates they can become a watcher of the repo on Github.
> > >>>
> > >>> 
> > >>> From: Paul Rogers <prog...@mapr.com>
> > >>> Sent: Thursday, October 19, 2017 9:43:26 AM
> > >>> To: dev@drill.apache.org
> > >>> Subject: Re: Excessive review comments
> > >>>
> > >>> Can we simply turn off the feature? I never, ever read the e-mails
> > coming
> > >>> from this source; I always follow the link back to the PR. Can we
> > reduce it
> > >>> to “Hey, just wanted to let you know that a new comment was posted.
> > Click
> > >>> _here_ to read it.”
> > >>>
> > >>> The only other solution is to give few review comments; not sure if
> we
> > >>> want to go that route...
> > >>>
> > >>> - Paul
> > >>>
> > >>>> On Oct 19, 2017, at 8:35 AM, Arina Yelchiyeva <
> > >>> arina.yelchiy...@gmail.com> wrote:
> > >>>>
> > >>>> Agree, I am not sure I saw this feature working as well.
>

Re: [DISCUSS] Drill 1.12.0 release

[Arina Yelchiyeva]

Current status:

Targeted for 1.12 release:
DRILL-5832: Migrate OperatorFixture to use SystemOptionManager rather than
mock (developer - Paul, code reviewer - ?) - waiting for the code review
DRILL-5842: Refactor and simplify the fragment, operator contexts for
testing (developer - Paul, code reviewer - ?) - waiting for the code review
DRILL-5834: Adding network functions (developer - Charles, code reviewer -
Arina) - waiting changes after code review
DRILL-5337: OpenTSDB plugin (developer - Dmitriy, code reviewer -
Arina) - waiting
for the code review
DRILL-5772: Enable UTF-8 support in query string by default (developer -
Arina, code reviewer - Paul) - finalizing approach
DRILL-4779: Kafka storage plugin support (developer - Anil, code reviewer -
?) - finishing implementation

Under question:
DRILL-4286: Graceful shutdown of drillbit (developer - Jyothsna, code
reviewer - ?) - waiting for the status update from the developer

Please free to suggest other items that are targeted for 1.12 release.
There are many Jiras that have fix version marked as 1.12, it would be good
if developers revisit them and update fix version to the actual one.
Link to the dashboard -
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185=DRILL=detail


Kind regards
Arina


On Wed, Oct 11, 2017 at 2:42 AM, Parth Chandra <par...@apache.org> wrote:

> I'm waiting to merge the SSL  changes in. Waiting a couple of days more to
> see if there are any more comments before I merge the changes in.
>
> On Mon, Oct 9, 2017 at 10:28 AM, Paul Rogers <prog...@mapr.com> wrote:
>
> > Hi Arina,
> >
> > In addition to my own PRs, there are several in the “active” queue that
> we
> > could get in if we can just push them over the line and clear the queue.
> > The owners of the PRs should check if we are waiting on them to take
> action.
> >
> > 977 DRILL-5849: Add freemarker lib to dependencyManagement to ensure
> > prop…
> > 976 DRILL-5797: Choose parquet reader from read columns
> > 975 DRILL-5743: Handling column family and column scan for hbase
> > 973 DRILL-5775: Select * query on a maprdb binary table fails
> > 972 DRILL-5838: Fix MaprDB filter pushdown for the case of nested
> > field (reg. of DRILL-4264)
> > 950 Drill 5431: SSL Support
> > 949 DRILL-5795: Parquet Filter push down at rowgroup level
> > 936 DRILL-5772: Add unit tests to indicate how utf-8 support can be
> > enabled / disabled in Drill
> > 904 DRILL-5717: change some date time test cases with specific
> > timezone or Local
> > 892 DRILL-5645: negation of expression causes null pointer exception
> > 889 DRILL-5691: enhance scalar sub queries checking for the cartesian
> > join
> >
> > (Items not on the list above have become “inactive” for a variety of
> > reasons.)
> >
> > Thanks,
> >
> > - Paul
> >
> > > On Oct 9, 2017, at 9:57 AM, Paul Rogers <prog...@mapr.com> wrote:
> > >
> > > Hi Arina,
> > >
> > > I’d like to include the following that are needed to finish up the
> > “managed” sort and spill-to-disk for hash agg:
> > >
> > > #928: DRILL-5716: Queue-driven memory allocation
> > > #958, DRILL-5808: Reduce memory allocator strictness for "managed"
> > operators
> > > #960, DRILL-5815: Option to set query memory as percent of total
> > >
> > > The following is needed to resolve issues with HBase support in empty
> > batches:
> > >
> > > #968, DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
> > >
> > > The following are nice-to-haves that build on work already done in this
> > release, and that some of my own work depends on:
> > >
> > > #970, DRILL-5832: Migrate OperatorFixture to use SystemOptionManager
> > rather than mock
> > > #978: DRILL-5842: Refactor and simplify the fragment, operator contexts
> > for testing
> > >
> > > The following is not needed for 1.12 per-se, but is the foundation for
> a
> > project I’m working on; would be good to get this in after 2-3 months of
> > review time:
> > >
> > > #921, foundation for batch size limitation
> > >
> > > The key issue with each of the above is that they each need a committer
> > to review. Some have reviews from non-committers. Any volunteers?
> > >
> > > Thanks,
> > >
> > > - Paul
> > >
> > >> On Oct 9, 2017, at 9:38 AM, Charles Givre <cgi...@gmail.com> wrote:
> > >>
> > >> Hi Arina,
> > >> I’d like to include Drill-5834, Adding network functions.
> 

Re: Excessive review comments

[Arina Yelchiyeva]

Agree, I am not sure I saw this feature working as well.
All it did it was sending all the emails at once, rather in the process of
comments emergence.

Kind regards
Arina

On Thu, Oct 19, 2017 at 6:27 PM, Julian Hyde  wrote:

> I don’t know whether anything is broken. I believed that the GitHub “start
> a review” feature would cause all review comments to be sent in a single
> email. But now I think of it, I’m not sure I ever saw it working. I wonder
> whether Github-ASF integration is at fault.
>
> Whatever the reasons for it, 39 emails to dev list is quite a blast.
> People tend to unsubscribe from lists if the volume is too high.
>
> Julian
>
>
> > On Oct 18, 2017, at 5:54 PM, Paul Rogers  wrote:
> >
> > With all due respect, I did start a review. Is something broken?
> >
> > - Paul
> >
> >> On Oct 18, 2017, at 3:36 PM, julianhyde  wrote:
> >>
> >> Github user julianhyde commented on a diff in the pull request:
> >>
> >>   https://github.com/apache/drill/pull/984#discussion_r145561518
> >>
> >>   --- Diff: 
> >> exec/java-exec/src/test/java/org/apache/drill/test/ClusterFixture.java
> ---
> >>   @@ -584,11 +492,14 @@ public static void defineWorkspace(Drillbit
> drillbit, String pluginName,
> >>  public static final String EXPLAIN_PLAN_TEXT = "text";
> >>  public static final String EXPLAIN_PLAN_JSON = "json";
> >>
> >>   -  public static FixtureBuilder builder() {
> >>   -FixtureBuilder builder = new FixtureBuilder()
> >>   +  public static FixtureBuilder builder(DirTestWatcher
> dirTestWatcher) {
> >>   --- End diff --
> >>
> >>   Jeez Paul, please start a review rather than making single review
> comments. I just got 39 emails from you, and so did everyone else on
> dev@drill.
> >>
> >>
> >> ---
> >
>
>

Re: Drill Hangout Jan 9, 2018

[Arina Yelchiyeva]

During hangout was decided to support JDK 8 by default in Apache Drill
1.13.0. We'l consider moving to Java 9 afterwards.
There is already open Jira for this issue -
https://issues.apache.org/jira/browse/DRILL-1491. I have linked all other
related issue to it as well.
Also it seems there is an Jira with open pull request to fix issues related
to JDK8 (https://issues.apache.org/jira/browse/DRILL-5730). Changes were
done by Tim.

Kind regards
Arina

On Tue, Jan 9, 2018 at 6:51 PM, Vlad Rozov <vro...@apache.org> wrote:

> Should we also talk about Java 9 and Hadoop 3.x support?
>
> Thank you,
>
> Vlad
>
>
> On 1/9/18 08:36, Arina Yelchiyeva wrote:
>
>> Mark Java 8 support as critical for 1.13.0 release?
>> For example, in Calcite 1.16 support for Java 7 will be dropped.
>>
>>
>> On Tue, Jan 9, 2018 at 9:34 AM, Aditya Allamraju <
>> aditya.allamr...@gmail.com
>>
>>> wrote:
>>> Hi Vlad,
>>>
>>> If this is open to everyone in this group, i would like to join this
>>> hangout.
>>> Please let me know.
>>>
>>> Thanks
>>> Aditya
>>>
>>> On Tue, Jan 9, 2018 at 1:22 AM, Vlad Rozov <vro...@apache.org> wrote:
>>>
>>> We will have our first bi-weekly hangout in the 2018 tomorrow at 10 am.
>>>> Please reply to this post with proposed topics to discuss.
>>>>
>>>> Hangout link: https://plus.google.com/hangouts/_/event/
>>>>
>>> ci4rdiju8bv04a64efj
>>>
>>>> 5fedd0lc
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>>
>

Re: Drill Hangout Jan 9, 2018

[Arina Yelchiyeva]

Mark Java 8 support as critical for 1.13.0 release?
For example, in Calcite 1.16 support for Java 7 will be dropped.


On Tue, Jan 9, 2018 at 9:34 AM, Aditya Allamraju  wrote:

> Hi Vlad,
>
> If this is open to everyone in this group, i would like to join this
> hangout.
> Please let me know.
>
> Thanks
> Aditya
>
> On Tue, Jan 9, 2018 at 1:22 AM, Vlad Rozov  wrote:
>
> > We will have our first bi-weekly hangout in the 2018 tomorrow at 10 am.
> > Please reply to this post with proposed topics to discuss.
> >
> > Hangout link: https://plus.google.com/hangouts/_/event/
> ci4rdiju8bv04a64efj
> > 5fedd0lc
> >
> > Thank you,
> >
> > Vlad
> >
>

Re: [ANNOUNCE] New PMC member: Paul Rogers

[Arina Yelchiyeva]

Congratulations, Paul!
Well deserved.

Kind regards
Arina

On Wed, Jan 31, 2018 at 10:13 AM, Robert Hou  wrote:

> Congratulations, Paul!
>
>
> --Robert
>
> 
> From: Abhishek Girish 
> Sent: Tuesday, January 30, 2018 9:31 PM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
>
> Congratulations, Paul!
>
> On Tue, Jan 30, 2018 at 2:48 PM, Sorabh Hamirwasia 
> wrote:
>
> > Congratulations Paul!
> >
> >
> > Thanks,
> > Sorabh
> >
> > 
> > From: AnilKumar B 
> > Sent: Tuesday, January 30, 2018 2:43:07 PM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
> >
> > Congratulations, Paul.
> >
> > Thanks & Regards,
> > B Anil Kumar.
> >
> > On Tue, Jan 30, 2018 at 2:34 PM, Chunhui Shi  wrote:
> >
> > > Congrats Paul! Well deserved!
> > >
> > > 
> > > From: Kunal Khatua 
> > > Sent: Tuesday, January 30, 2018 2:05:56 PM
> > > To: dev@drill.apache.org
> > > Subject: RE: [ANNOUNCE] New PMC member: Paul Rogers
> > >
> > > Congratulations, Paul !
> > >
> > > -Original Message-
> > > From: salim achouche [mailto:sachouc...@gmail.com]
> > > Sent: Tuesday, January 30, 2018 2:00 PM
> > > To: dev@drill.apache.org; Padma Penumarthy 
> > > Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
> > >
> > > Congrats Paul!
> > >
> > > Regards,
> > > Salim
> > >
> > > > On Jan 30, 2018, at 1:58 PM, Padma Penumarthy 
> > > wrote:
> > > >
> > > > Congratulations Paul.
> > > >
> > > > Thanks
> > > > Padma
> > > >
> > > >
> > > >> On Jan 30, 2018, at 1:55 PM, Gautam Parai  wrote:
> > > >>
> > > >> Congratulations Paul!
> > > >>
> > > >> 
> > > >> From: Timothy Farkas 
> > > >> Sent: Tuesday, January 30, 2018 1:54:43 PM
> > > >> To: dev@drill.apache.org
> > > >> Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
> > > >>
> > > >> Congrats!
> > > >>
> > > >> 
> > > >> From: Aman Sinha 
> > > >> Sent: Tuesday, January 30, 2018 1:50:07 PM
> > > >> To: dev@drill.apache.org
> > > >> Subject: [ANNOUNCE] New PMC member: Paul Rogers
> > > >>
> > > >> I am pleased to announce that Drill PMC invited Paul Rogers to the
> > > >> PMC and he has accepted the invitation.
> > > >>
> > > >> Congratulations Paul and thanks for your contributions !
> > > >>
> > > >> -Aman
> > > >> (on behalf of Drill PMC)
> > > >
> > >
> > >
> >
>

Re: [VOTE] Apache Drill release 1.14.0 - RC1

[Arina Yelchiyeva]

Hi Boaz,

it's unfortunate we have to have second iteration of RC. New RC will
contain two commits (one from previous iteration) from:
https://github.com/apache/drill/pull/1404
https://github.com/apache/drill/pull/1406

But I believe we should merge them into master first before creating new
RC. Both has passed CR, so it's OK to merge them.

Kind regards,
Arina

On Mon, Jul 30, 2018 at 11:48 PM Boaz Ben-Zvi  wrote:

>OK -- RC1 is off; will produce RC2 soon with the PR #1406 (and thanks
> Vitalii for the other comments)
>
>   Boaz
>
>
> On 7/30/18 9:53 AM, Charles Givre wrote:
> > I attempted to build from source and got the following errors:
> >
> > Results :
> >
> > Failed tests:
> >
> TestConvertCountToDirectScan.ensureCorrectCountWithMissingStatistics:153->PlanTestBase.testPlanMatchingPatterns:84->PlanTestBase.testPlanMatchingPatterns:103
>
> > Found unwanted pattern in plan: DynamicPojoRecordReader
> > 00-00Screen
> > 00-01 Project(cnt_str=[$0], cnt_total=[$1])
> > 00-02 Scan(groupscan=[files =
> >
> [/Users/cgivre/github/drill-dev/rc1/apache-drill-1.14.0-src/exec/java-exec/target/org.apache.drill.exec.planner.logical.TestConvertCountToDirectScan/dfsTestTmp/1532967947836-0/wide_str_table/0_0_1.parquet,
>
> >
> /Users/cgivre/github/drill-dev/rc1/apache-drill-1.14.0-src/exec/java-exec/target/org.apache.drill.exec.planner.logical.TestConvertCountToDirectScan/dfsTestTmp/1532967947836-0/wide_str_table/0_0_2.parquet],
>
> > numFiles = 2, DynamicPojoRecordReader{records = [[0, 2]]}])
> >
> >
> > Tests run: 3331, Failures: 1, Errors: 0, Skipped: 156
> >
> >
> >
> >
> >> On Jul 30, 2018, at 11:17, Vitalii Diravka  >> > wrote:
> >>
> >> Hi all!
> >>
> >> I'm in the process of verifying the Drill RC1.
> >> And I always get BUILD FAILURE for "mvn clean install". It is related to
> >> DRILL-6641 (PR is on review).
> >> It fails randomly from Jira description, but for me the test fails every
> >> time.
> >> Does anybody else have this issue? Or maybe it is my local environment
> >> setup causes it.
> >> I want to be sure, since I can not validate RC1 without clean build with
> >> unit tests.
> >>
> >> Boaz, could you please edit the [1] link, I assume it should be
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12342097
> >> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_ReleaseNote.jspa-3FprojectId-3D12313820-26version-3D12342097=DwMFAg=cskdkSMqhcnjZxdQVpwTXg=7lXQnf0aC8VQ0iMXwVgNHw=fsrZudttKIbOzqwzR6E8VlkHuGlqEq1RLobDOm8zivw=b4JQ8vrZwv1C5l6cEU2W4Al-Aiw7XHkz1RMlAUtHSec=
> >
> >> and add the space after the first link in [4].
> >>
> >> Kind regards
> >> Vitalii
> >>
> >>
> >> On Sat, Jul 28, 2018 at 2:25 AM Boaz Ben-Zvi  wrote:
> >>
> >>>  Hi Drillers,
> >>>
> >>> Proposing the *second* Release Candidate (RC1) for the Apache
> >>> Drill, version 1.14.0 .
> >>>
> >>> This RC1 includes 234 committed Jiras [1]. Thanks to all the Drill
> >>> developers who works hard and contributed to this release.
> >>>
> >>> The RC1 tarballs are hosted at [2] , and the Maven artifacts are at
> [3].
> >>>
> >>> This Release Candidate is based on (Apache Drill branch named "1.14.0")
> >>> commit ID: c705271d550a6adf0a874cd4a6bddd62d5ecc1d9 , available at [4].
> >>>
> >>> Please download and try/test this Release Candidate.
> >>>
> >>> Given that our bylaws require 3 business days, the vote would end on
> >>> Wednesday, August 1st, 2018 at 5:00 PM PDT .
> >>>
> >>> [ ] +1
> >>> [ ] +0
> >>> [ ] -1
> >>>
> >>>  My vote is +1 !!
> >>>
> >>>Thank you,
> >>>
> >>> Boaz
> >>>
> >>> [1] https://issues.apache.org/jira/browse/DRILL-6637?filter=12344431
> >>>
> >>> [2] http://home.apache.org/~boaz/drill/releases/1.14.0/rc1/
> >>>
> >>> [3]
> >>> https://repository.apache.org/content/repositories/orgapachedrill-1049
> >>>
> >>> [4] https://github.com/apache/drill/tree/1.14.0OR
> >>> https://github.com/Ben-Zvi/drill/tree/drill-1.14.0 (commit id:
> >>> 65a6cb5233058b24613aaecfa9c9c7007a12c7e8)
> >>>
> >>>
> >>>
> >
>
>

Re: July Apache Drill board report

[Arina Yelchiyeva]

Sure, Charles. I have added point about newly added functions as well.
We have so many features / improvements so it's hard to list them all :),
that's why also added note in the report that the list is not full.

=

## Description:
 - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
Storage.

## Issues:
 - There are no issues requiring board attention at this time.

## Activity:
 - Currently 1.14.0 release is being prepared. Below is not a full list of
features / enhancements besides numerous bug fixes to be included in the
release:
  - Decimal data type enhancements
  - Generic Logfile Format Plugin
  - Image Metadata Format Plugin
  - Official Drill Docker Container
  - Spill to disk for the Hash Join implementation
  - Drill Plugins Handler
  - Support CGROUPs resource management
  - Support Phonetic and String Distance functions

## Health report:
 - The project is healthy. Development activity as reflected in the pull
requests and JIRAs is good.
 - Activity on the dev and user mailing lists are stable.
 - Two new committers and one new PMC member were added in the last period.

## PMC changes:

 - Currently 20 PMC members.
 - Vitalii Diravka was added to the PMC on Tue Jun 26 2018
 - Arina Ielchiieva was elected as new PMC Chair on Wed Jul 30 2018

## Committer base changes:

 - Currently 45 committers.
 - New commmitters:
- Padma Penumarthy was added as a committer on Mon Jun 18 2018
- Timothy Farkas was added as a committer on Fri May 25 2018

## Releases:

 - Last release was 1.13.0 on Mon Mar 19 2018

## Mailing list activity:

 - dev@drill.apache.org:
- 432 subscribers (down -5 in the last 3 months):
- 2412 emails sent to list (2450 in previous quarter)

 - iss...@drill.apache.org:
- 19 subscribers (up 0 in the last 3 months):
- 5094 emails sent to list (3320 in previous quarter)

 - u...@drill.apache.org:
- 605 subscribers (up 0 in the last 3 months):
- 369 emails sent to list (346 in previous quarter)


## JIRA activity:

 - 277 JIRA tickets created in the last 3 months
 - 239 JIRA tickets closed/resolved in the last 3 months

=



On Tue, Jul 31, 2018 at 2:45 PM Charles Givre  wrote:

> Hello Arina,
> Would you want to put something about the Phonetic and String Distance
> enhancements as well for Drill 1.14?
>
> > On Jul 31, 2018, at 07:43, Arina Ielchiieva  wrote:
> >
> > Hello all,
> >
> > please take a look at the draft board report for this quarter and let me
> > know if you have any comments.
> >
> > Thanks,
> > Arina
> >
> > =
> >
> > ## Description:
> > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> > Storage.
> >
> > ## Issues:
> > - There are no issues requiring board attention at this time.
> >
> > ## Activity:
> > - Currently 1.14.0 release is being prepared. It includes the following
> > features / enhancements along with bug fixes:
> >  - Decimal data type enhancements
> >  - Generic Logfile Format Plugin
> >  - Image Metadata Format Plugin
> >  - Official Drill Docker Container
> >  - Implement spill to disk for the Hash Join
> >  - Drill Plugins Handler
> >  - Support CGROUPs resource management
> >
> > ## Health report:
> > - The project is healthy. Development activity as reflected in the pull
> > requests and JIRAs is good.
> > - Activity on the dev and user mailing lists are stable.
> > - Two new committers and one new PMC member were added in the last
> period.
> >
> > ## PMC changes:
> >
> > - Currently 20 PMC members.
> > - Vitalii Diravka was added to the PMC on Tue Jun 26 2018
> > - Arina Ielchiieva was elected as new PMC Chair on Wed Jul 30 2018
> >
> > ## Committer base changes:
> >
> > - Currently 45 committers.
> > - New commmitters:
> >- Padma Penumarthy was added as a committer on Mon Jun 18 2018
> >- Timothy Farkas was added as a committer on Fri May 25 2018
> >
> > ## Releases:
> >
> > - Last release was 1.13.0 on Mon Mar 19 2018
> >
> > ## Mailing list activity:
> >
> > - dev@drill.apache.org:
> >- 432 subscribers (down -5 in the last 3 months):
> >- 2412 emails sent to list (2450 in previous quarter)
> >
> > - iss...@drill.apache.org:
> >- 19 subscribers (up 0 in the last 3 months):
> >- 5094 emails sent to list (3320 in previous quarter)
> >
> > - u...@drill.apache.org:
> >- 605 subscribers (up 0 in the last 3 months):
> >- 369 emails sent to list (346 in previous quarter)
> >
> >
> > ## JIRA activity:
> >
> > - 277 JIRA tickets created in the last 3 months
> > - 239 JIRA tickets closed/resolved in the last 3 months
> >
> > =
>
>

Re: [VOTE] Apache Drill release 1.14.0 - RC1

[Arina Yelchiyeva]

Please disregard this email, found new email thread for RC2.

On Tue, Jul 31, 2018 at 7:30 PM Arina Ielchiieva  wrote:

> Used the following links for the verification, since last email does not
> provide any.
>
> https://repository.apache.org/content/repositories/orgapachedrill-1050/
> https://github.com/apache/drill/tree/1.14.0
> http://home.apache.org/~boaz/drill/releases/1.14.0/rc2/
>
> Built from source, ran unit tests, also Functional and Advanced tests.
> Downloaded tar.gz and started Drill in embedded mode on Windows.
> On both installations ran random queries against system tables and files
> (mostly long-running queries).
> Also checked Web UI (plugins export, jstack copy, profiles display,
> exceptions display for invalid queries).
>
> +1 (binding)
>
> Kind regards,
> Arina
>
> On Tue, Jul 31, 2018 at 1:02 AM Boaz Ben-Zvi  wrote:
>
>>  Hi Arina,
>>
>> The RC1 (and now RC2) are being built in a separate branch - 1.14.0 ;
>> building the RC.. over the master would require a forced push (as the top
>> of master is marked 1.15.0-SNAPSHOT).
>>
>> We can just work by cherry-picking the needed commits into the 1.14.0
>> branch. It is about the same (other than different commit IDs).
>>
>> The master branch has been open for new work.
>>
>>Thanks,
>>
>> Boaz
>>
>> On 7/30/18 1:56 PM, Arina Yelchiyeva wrote:
>>
>> Hi Boaz,
>>
>> it's unfortunate we have to have second iteration of RC. New RC will
>> contain two commits (one from previous iteration) from:
>> https://github.com/apache/drill/pull/1404
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_drill_pull_1404=DwMFaQ=cskdkSMqhcnjZxdQVpwTXg=7lXQnf0aC8VQ0iMXwVgNHw=Y5ocpPz6MxuOl_b8KpxBSxcQNRR3fYti88mdkysBYt4=-5i5DCVG_AdKnSWwDURr97RbD26zF2dyu0q7Tm4I9vk=>
>> https://github.com/apache/drill/pull/1406
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_drill_pull_1406=DwMFaQ=cskdkSMqhcnjZxdQVpwTXg=7lXQnf0aC8VQ0iMXwVgNHw=Y5ocpPz6MxuOl_b8KpxBSxcQNRR3fYti88mdkysBYt4=_UQTeIIMlX9tdjac2_FAano7MB27pbiooXnoR1xMu14=>
>>
>> But I believe we should merge them into master first before creating new
>> RC. Both has passed CR, so it's OK to merge them.
>>
>> Kind regards,
>> Arina
>>
>> On Mon, Jul 30, 2018 at 11:48 PM Boaz Ben-Zvi  wrote:
>>
>>>OK -- RC1 is off; will produce RC2 soon with the PR #1406 (and thanks
>>> Vitalii for the other comments)
>>>
>>>   Boaz
>>>
>>>
>>> On 7/30/18 9:53 AM, Charles Givre wrote:
>>> > I attempted to build from source and got the following errors:
>>> >
>>> > Results :
>>> >
>>> > Failed tests:
>>> >
>>> TestConvertCountToDirectScan.ensureCorrectCountWithMissingStatistics:153->PlanTestBase.testPlanMatchingPatterns:84->PlanTestBase.testPlanMatchingPatterns:103
>>>
>>> > Found unwanted pattern in plan: DynamicPojoRecordReader
>>> > 00-00Screen
>>> > 00-01 Project(cnt_str=[$0], cnt_total=[$1])
>>> > 00-02 Scan(groupscan=[files =
>>> >
>>> [/Users/cgivre/github/drill-dev/rc1/apache-drill-1.14.0-src/exec/java-exec/target/org.apache.drill.exec.planner.logical.TestConvertCountToDirectScan/dfsTestTmp/1532967947836-0/wide_str_table/0_0_1.parquet,
>>>
>>> >
>>> /Users/cgivre/github/drill-dev/rc1/apache-drill-1.14.0-src/exec/java-exec/target/org.apache.drill.exec.planner.logical.TestConvertCountToDirectScan/dfsTestTmp/1532967947836-0/wide_str_table/0_0_2.parquet],
>>>
>>> > numFiles = 2, DynamicPojoRecordReader{records = [[0, 2]]}])
>>> >
>>> >
>>> > Tests run: 3331, Failures: 1, Errors: 0, Skipped: 156
>>> >
>>> >
>>> >
>>> >
>>> >> On Jul 30, 2018, at 11:17, Vitalii Diravka >> >> <mailto:vitalii.dira...@gmail.com>> wrote:
>>> >>
>>> >> Hi all!
>>> >>
>>> >> I'm in the process of verifying the Drill RC1.
>>> >> And I always get BUILD FAILURE for "mvn clean install". It is related
>>> to
>>> >> DRILL-6641 (PR is on review).
>>> >> It fails randomly from Jira description, but for me the test fails
>>> every
>>> >> time.
>>> >> Does anybody else have this issue? Or maybe it is my local environment
>>> >> setup causes it.
>>> >> I want to be sure, since I can not validate RC1 without clean build
>>> with
>

[DISCUSS] sqlline upgrade

[Arina Yelchiyeva]

Hi all,

currently Apache Drill is using custom version of Sqlline [1]. It has some
custom fixes, the importance of which I propose to discuss.

*1. Isolation.*
Drill supports TRANSACTION_NONE only, default in Sqlline
is TRANSACTION_REPEATABLE_READ. Apparently the was not possibility to
override the default or by any other reason, in custom version setting
transaction is commented out. After upgrade to the latest version, during
connection Drill errors that default transaction level is not acceptable
but this can be easily fixed by passing arguments to Sqlline to change the
defaults. Example will be provided below.

*2.  Resizing of output.*
After upgrade to the latest version, output is weirdly resized, custom
version has ResizingRowsProvider which fixed this issue but now this can be
easily fixed by passing incremental=false when calling Sqlline.

Example for points 1 and 2:
CMD="$JAVA $SHELL_OPTS -cp $CP sqlline.SqlLine -d
org.apache.drill.jdbc.Driver --maxWidth=1* --isolation=TRANSACTION_NONE
--incremental=false*"

I haven't noticed any other issues with Sqlline that might regress after
the upgrade. If I have missed something else, please feel free to correct
me.

*3. Output of Drill version at start up.*
Now:
*apache drill 1.15.0-SNAPSHOT *
*"got drill?"*

After the upgrade:
*sqlline version 1.4.0*
*0: jdbc:drill:zk=local>*

*4. Options that Drill did not support were commented out, so they are not
displayed in help menu.*

If we upgrade, we'll lose the last two enhancements, though I don't think
they are crucial. Other projects like Apache Phoenix are doing fine without
them. Plus I think it's quite obvious why moving from custom version is a
good choice.

Any thoughts?

[1] https://github.com/mapr/sqlline/commits/1.1.9-drill-r7

Kind regards,
Arina

Re: [DISCUSSION] current project state

[Arina Yelchiyeva]

1. Regarding Drill metastore, its under investigation, please follow up
with DRILL-6552.
2. UDFs: I would not say, it's that quit to write UDFs in Drill.
Definitely, it could have been done easier but even for current state we
have good manuals. Regarding adding support for different languages like
python, that would require full re-write on UDFs code handling, since Drill
heavily relies on Java source code when during UDFs initialization. Though
generally it's a good idea since, Hive, for example, supports Scala, Python
for UDFs.
3. Drill vs Arrow is the topic I heard since I have started working with
Drill. But so far nobody dared to tackle it. I would suspect Drill first
would have to contribute changes in Arrow to be able to migrate which could
be a show-stopper if Arrow community does not accept them.

On Tue, Aug 14, 2018 at 6:37 AM Charles Givre  wrote:

> I’d like to weigh in here as well. As a long time user of Drill, I really
> would like to see more people using it and I think there are a few key
> aspects that could really help on that front.
>
> The first of which is the Arrow integration.  I’m not enough of a software
> engineer to understand all the internal details here, but as I understand
> it, the promise of Arrow is that many tools will share a common memory
> model and that it will be possible to transfer data from one tool to the
> other without having to serialize/deserialize the data.  In the data
> science community many of the major platforms, Python-pandas, R, and Spark
> are moving or have adopted Arrow.
> Drill’s strength is the ease that it can query many different data sources
> and if Drill were to adopt Arrow, I suspect that many people would adopt it
> as a part of a machine learning pipeline.  Just recently, I attempted to do
> some data manipulation using Spark, and couldn’t help but notice how
> difficult ti was in contrast with Drill. I’m sure this is a very complex
> task, but I do think that it could be worth it in the end.
>
> Secondly, I’d like to second Paul’s call to simplify the interfaces for
> UDFs, Format and ideally storage plugins.  A core strength of Drill is its
> extensibility and making it easier would be a great thing.  I was wondering
> whether it would be possible or even a good idea, to enable users to write
> UDFs in a scripting language such as python.
>
> Thirdly,
> i would really like to see us add more functionality to Drill.  @Arina,
> your work to build a storage plugin for ElasticSearch is really great and I
> think more capabilities like that are really needed.  I’d like to see a
> generic HTTP storage plugin, a storage plugin for Google Sheets,  If I can
> figure out how storage plugins work, I’ll gladly work on some of these.
>
> Just my .02.
> — C
>
>
>
>
>
> > On Aug 13, 2018, at 21:21, Paul Rogers 
> wrote:
> >
> > Hi Arina,
> >
> > Another topic would be whether/how to round out Drill's data model.
> Drill's scalar and nullable types are pretty solid. Great work was done
> recently for Decimal (though the old types still remain.) Good support is
> now available for nested types to do implicit joins to produce SQL-friendly
> flat records.
> > But, opportunities for improvement still remain. Date/Time has timezone
> issues. Union, List and Repeated List never quite worked. There are a few
> types identified in the code, but not implemented (dates with TZ, tiny
> ints, etc.) How should Drill bridge. the gap from arrays and maps (really,
> structs) on the one hand, and plain-old-relational ODBC/JDBC/BI tools on
> the other?
> >
> > Would be good to finalize the data types and their mapping to plain SQL:
> either keep a type and make it fully work if it has holes, or drop it.
> Unions and Lists are the messiest. They are incomplete in part, because
> they are trying to do the impossible: to predict the future well enough
> that Drill can handle columns with varying or ambiguous data types (that
> is, to handle schema changes.) Is there a better way to handle this issue
> (such as with metadata hints)? That is, rather than fight with conflicting
> types at run time, simply declare the common type in metadata so all
> operators and record batches agree on the type.
> >
> > And, of course, there is the lingering issue of Drill vectors vs. Arrow.
> Arrow did great work in metadata, but seems to have kept some of the
> awkward aspects of Drill's original memory model (lack of control over
> batch sizes, ability to fragment memory.) Might there be a resyncing of the
> two projects: Drill picks up Arrow's metadata and APIs, Arrow picks up
> Drill's memory improvements, such as the size-limiting "result set loader"
> framework.
> >
> > Big-picture issues such as this tend to get lost in the 2270 open Jira
> tickets. How might the project create some "theme" tickets (or Wiki pages
> or whatever) to help pull the main issues out of the wealth of detail in
> Jira?
> >
> > Thanks,
> > - Paul
> >
> >
> >
> >On Monday, August 13, 2018, 11:07:39 AM 

Re: [DISCUSS] sqlline upgrade

[Arina Yelchiyeva]

@John Omernik 
if I understood you correctly, new options were going to be added in
existing scripts, so users would have been calling, for example,
`drill-embedded` as usual.

Based on the feedback before moving to the new Sqlline version we would
expect the following to be fixed:
1. Customization in sqlline (
https://github.com/julianhyde/sqlline/issues/106).
2. Upgrade to jline3 (https://github.com/julianhyde/sqlline/issues/73)
[took a quick look, this one would require some re-write in sqlline,
packages, interfaces etc have changed]

@Parth Chandra 
could you please give an example about
{quote}
While we are at it, we could fix the (very annoying) behavior of SQLLine
where it tries to disconnect from the database even after the connect call
failed.
{quote}

I have updated https://issues.apache.org/jira/browse/DRILL-3853 with
relevant information.

Kind regards,
Arina



On Tue, Aug 14, 2018 at 2:14 AM Parth Chandra  wrote:

> #3 was considered a bit of a requirement when it was done. Having SQLLine
> show a different version from that of Drill caused confusion among users.
> #4 seems another usability issue. I would feel less confident in a product
> whose command line tool shows me commands that are not supported.
> Also you will lose the somewhat amusing Drill quotes.
>
> So what is in the latest release of SQLLine that we do not have in the
> Drill version?  (Not much from what I can see).
>
> It would be nice to externalize the changes made in the Drill version and
> submit the changes back to SQLLine.
> We could move the options supported, the default startup options, the
> product version, and the startup quotes to an external resource file that
> is optionally passed to SQLLine at startup.
> While we are at it, we could fix the (very annoying) behavior of SQLLine
> where it tries to disconnect from the database even after the connect call
> failed.
>
> Note that SQLLine is not actively being updated by a team like many other
> open source projects are, so if we want scrolling thru multi line sql
> statements, then we probably should do it ourselves. (Probably needs an
> update to JLine 3.0.2 or newer. see :
> https://github.com/jline/jline3/issues/36)
>
>
>
>
> On Mon, Aug 13, 2018 at 4:00 PM, Boaz Ben-Zvi  wrote:
>
> > supports scrolling through multi-line SQL statements as single snippets
> in
> >>> history
> >>>
> >>
> >   I opened an issue (#73) for that back in April (
> > https://github.com/julianhyde/sqlline/issues/73 ), but no progress so
> > far
> >
> >
> >
> > On 8/13/18 10:15 AM, Abhishek Girish wrote:
> >
> >> +1. Need to try out (2) to understand it's impact on usability. Okay
> with
> >> (3). Not sure of consequences of (4) - what will happen to options Drill
> >> doesn't support?
> >>
> >> Also, do we know if the latest sqlline supports scrolling through
> >> multi-line SQL statements as single snippets in history?
> >>
> >> On Mon, Aug 13, 2018 at 10:08 AM Arina Yelchiyeva <
> >> arina.yelchiy...@gmail.com> wrote:
> >>
> >> Hi all,
> >>>
> >>> currently Apache Drill is using custom version of Sqlline [1]. It has
> >>> some
> >>> custom fixes, the importance of which I propose to discuss.
> >>>
> >>> *1. Isolation.*
> >>> Drill supports TRANSACTION_NONE only, default in Sqlline
> >>> is TRANSACTION_REPEATABLE_READ. Apparently the was not possibility to
> >>> override the default or by any other reason, in custom version setting
> >>> transaction is commented out. After upgrade to the latest version,
> during
> >>> connection Drill errors that default transaction level is not
> acceptable
> >>> but this can be easily fixed by passing arguments to Sqlline to change
> >>> the
> >>> defaults. Example will be provided below.
> >>>
> >>> *2.  Resizing of output.*
> >>> After upgrade to the latest version, output is weirdly resized, custom
> >>> version has ResizingRowsProvider which fixed this issue but now this
> can
> >>> be
> >>> easily fixed by passing incremental=false when calling Sqlline.
> >>>
> >>> Example for points 1 and 2:
> >>> CMD="$JAVA $SHELL_OPTS -cp $CP sqlline.SqlLine -d
> >>> org.apache.drill.jdbc.Driver --maxWidth=1*
> >>> --isolation=TRANSACTION_NONE
> >>> --incremental=false*"
> >>>
> >>> I haven't noticed any other issues with Sqlline that might regress
> after
> >>> the upgrade. If I have missed something