Re: [Question] Using loggers in tests

[Timothy Farkas]

Thanks Paul, forgot about that. I'll migrate all the tests off of 
System.out.print and onto LogFixture, I don't think its worth it to create a 
TestLogger.out since we should all be using loggers anyway. I'll also add a 
checkstyle check that will cause the build to fail of System.out.print is used 
anywhere.

Thanks,
Tim


From: Paul Rogers 
Sent: Thursday, May 24, 2018 7:08:28 PM
To: dev@drill.apache.org
Subject: Re: [Question] Using loggers in tests

LogFixture? As illustrated in ExampleTest?

This fixture lets you turn on all or selected loggers for the duration of a 
single test. I used it all the time when debugging. Works great.
It works when turning loggers on when the default is that they are off. For 
whatever reason, I could not get it to work to turn off logging that was 
enabled in the config file.
At one point I did look into adding a custom debug logger that works just like 
System.out, but is disabled by default. That way, conversion was just a matter 
of replacing System.out with TestLogger.out which can be done via 
search/replace. Not sure if I ever checked that in, but it would be trivial to 
replicate.
Thanks,
- Paul



On Thursday, May 24, 2018, 5:32:49 PM PDT, Timothy Farkas 
 wrote:

 Hi All,

I was wondering if there was a magical way to enable the Slf4j loggers for unit 
tests without adding a logback-test.xml file into src/test/resources for a 
submodule in the project? If not, would there by any issues with adding a 
default logback-test.xml file that has logging disabled by default to each 
submodule's src/test/resources directory? I'd like to do this in order to 
discourage the use of System.out.println in tests (and eventually prohibit it 
completely) by providing an easy to use out of the box alternative. Currently 
our test logs are polluted by many System.out.print statements and switching to 
using logging will allow us to have our test messages when we want them, and to 
disable them when we don't want them.

Thanks,
Tim

Re: [Question] Using loggers in tests

[Paul Rogers]

LogFixture? As illustrated in ExampleTest?

This fixture lets you turn on all or selected loggers for the duration of a 
single test. I used it all the time when debugging. Works great.
It works when turning loggers on when the default is that they are off. For 
whatever reason, I could not get it to work to turn off logging that was 
enabled in the config file.
At one point I did look into adding a custom debug logger that works just like 
System.out, but is disabled by default. That way, conversion was just a matter 
of replacing System.out with TestLogger.out which can be done via 
search/replace. Not sure if I ever checked that in, but it would be trivial to 
replicate.
Thanks,
- Paul

 

On Thursday, May 24, 2018, 5:32:49 PM PDT, Timothy Farkas 
 wrote:  
 
 Hi All,

I was wondering if there was a magical way to enable the Slf4j loggers for unit 
tests without adding a logback-test.xml file into src/test/resources for a 
submodule in the project? If not, would there by any issues with adding a 
default logback-test.xml file that has logging disabled by default to each 
submodule's src/test/resources directory? I'd like to do this in order to 
discourage the use of System.out.println in tests (and eventually prohibit it 
completely) by providing an easy to use out of the box alternative. Currently 
our test logs are polluted by many System.out.print statements and switching to 
using logging will allow us to have our test messages when we want them, and to 
disable them when we don't want them.

Thanks,
Tim  

[Question] Using loggers in tests

[Timothy Farkas]

Hi All,

I was wondering if there was a magical way to enable the Slf4j loggers for unit 
tests without adding a logback-test.xml file into src/test/resources for a 
submodule in the project? If not, would there by any issues with adding a 
default logback-test.xml file that has logging disabled by default to each 
submodule's src/test/resources directory? I'd like to do this in order to 
discourage the use of System.out.println in tests (and eventually prohibit it 
completely) by providing an easy to use out of the box alternative. Currently 
our test logs are polluted by many System.out.print statements and switching to 
using logging will allow us to have our test messages when we want them, and to 
disable them when we don't want them.

Thanks,
Tim

Re: Delete spurious branches from Apache

[Parth Chandra]

I believe Drill web site and documentation is generated automatically from
the gh_pages. Moving this is probably not advisable.

On Tue, May 22, 2018 at 3:14 PM, Boaz Ben-Zvi  wrote:

>   Done – all three were deleted, using github (thanks @Vlad Rozov).
>
> There is also an active branch called “gh-pages” (and a non-active one
> called “gh-pages-master”) that seems to be used for the Drill documentation.
> Maybe this work should be moved to its own Repo ?  It does not make sense
> to be a part of the Drill sources (e.g., note this branch is “3086 commits
> behind”).
> @Bridget Bevens – what do you think ?
>
> Thanks,
>
>Boaz
>
> On 5/22/18, 1:02 PM, "Parth Chandra"  wrote:
>
> Yes, please go ahead and remove these branches.
>
> On Mon, May 21, 2018 at 8:06 PM, Vlad Rozov  wrote:
>
> > There is an option  to delete the branch on github directly (no need
> to
> > use "git push").
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 5/21/18 18:35, Boaz Ben-Zvi wrote:
> >
> >> I mistakenly pushed a branch (“MERGE-180521-01”) into the Apache
> >> repo, and plan to delete it soon
> >> (i.e. do  “git push https://urldefense.proofpoint.
> com/v2/url?u=https-3A__github.com_apache_drill.git=DwIFaQ&
> c=cskdkSMqhcnjZxdQVpwTXg=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU=
> Jgwab6ezufiqEbsi7h1wNNGZPnUbR28Caq2JWPsSlp0=
> JiRFn2vcSJHxPA09nlPxBKSi9xk0thr-JKiaYNyMEws= --delete
> >> MERGE-180521-01” ).
> >>
> >>Just in case someone notices 
> >>
> >>On this occasion: Looks like there are other similar such
> branches:
> >> “DRILL-3478” and “DRILL-4235” ; any objection to deleting those as
> well ?
> >>
> >>  Thanks,
> >>
> >>  Boaz
> >>
> >>
> >>
> >
>
>
>

[jira] [Created] (DRILL-6445) Fix existing test cases in TestScripts.java and add new test case for DRILLBIT_CONTEXT variable

[Sorabh Hamirwasia (JIRA)]

Sorabh Hamirwasia created DRILL-6445:


 Summary: Fix existing test cases in TestScripts.java and add new 
test case for DRILLBIT_CONTEXT variable
 Key: DRILL-6445
 URL: https://issues.apache.org/jira/browse/DRILL-6445
 Project: Apache Drill
  Issue Type: Task
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
 Fix For: 1.14.0


Under drill-yarn module there is [TestScripts.java 
file|https://github.com/apache/drill/blob/master/drill-yarn/src/test/java/org/apache/drill/yarn/scripts/TestScripts.java]
 created for testing the scripts provided by Drill to setup the environment. 
Currently those tests are failing. This Jira is to make sure all the tests are 
passing and few new tests are added for DRILLBIT_CONTEXT variable inside the 
script.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6444) Hash Join: Avoid partitioning when memory is sufficient

[Boaz Ben-Zvi (JIRA)]

Boaz Ben-Zvi created DRILL-6444:
---

 Summary: Hash Join: Avoid partitioning when memory is sufficient 
 Key: DRILL-6444
 URL: https://issues.apache.org/jira/browse/DRILL-6444
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Relational Operators
Reporter: Boaz Ben-Zvi
Assignee: Boaz Ben-Zvi


The Hash Join Spilling feature introduced partitioning (of the incoming build 
side) which adds some overhead (copying the incoming data, row by row). That 
happens even when no spilling is needed.

Suggested optimization: Try reading the incoming build data without 
partitioning, while checking that enough memory is available. In case the whole 
build side (plus hash table) fits in memory - then continue like a "single 
partition". In case not, then need to partition the data read so far and 
continue as usual (with partitions).

(See optimization 8.1 in the Hash Join Spill design document: 
[https://docs.google.com/document/d/1-c_oGQY4E5d58qJYv_zc7ka834hSaB3wDQwqKcMoSAI/edit]
 )

This is currently implemented only for the case of num_partitions = 1 (i.e, no 
spilling, and no memory checking).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)