Re: Debugging Drill in Eclipse

[Paul Rogers]

Thanks for the hint about configuring Eclipse to launch Drill. That would be 
handy for when we need an actual Drillbit process. The technique I mentioned 
involves running Drill in embedded mode, often as part of test case. The 
“cluster test” framework makes that very simple and fast. Combined, we have 
tools to be able to run in whatever mode is easiest for a given test.

I gathered some existing notes into a more complete description of the test 
framework: [1]

This is a starter set. Please let me know if there are particular areas that 
need more detail. I’m hoping that once the Wiki page gets you started, that the 
detailed Javadoc in the various test classes will help you tackle the more 
complex cases. But, do let me know if more description is helpful.

Thanks,

- Paul

[1] https://github.com/paul-rogers/drill/wiki/Cluster-Fixture-Framework

> On Jul 14, 2017, at 8:23 AM, Muhammad Gelbana  wrote:
> 
> When you said I'm rebuilding Drill, do you mean I'm running "*mvn clean
> install*" after every change ? Actually I'm configuring eclipse to run the
> same java command that "*drillbit.sh start*" command would run.
> 
> But I have to say that your method is much more efficient and practical. I
> should've used test cases for development from the start.
> 
> I recently started exploring the test cases classes but I found it very
> hard to comprehend so If there is some sort of a guide to the testing
> classes architecture, packages structure\categorization..etc, it would be
> very helpful if you share it. I looked into your github wiki
>  but I didn't find
> what I need.
> 
> 
> 
> - Gelbana
> 
> On Thu, Jul 13, 2017 at 6:53 PM, Paul Rogers  wrote:
> 
>> Hi Muhammad,
>> 
>> There are several issues here.
>> 
>> First, the problem you describe is not related to Eclipse. Second, several
>> of us do use Eclipse and it works fine. Third, there are far easier ways to
>> debug Drill in Eclipse then building Drill and doing remote debugging.
>> 
>> First the error. The problem is that some bit of code is referring to a
>> config parameter that does not appear in the config files. This kind of key
>> would normally appear in some drill-module.conf file. (The file
>> drill-override.conf is only for the very few config setting that you must
>> modify for your site.)
>> 
>> My source is a bit different than your, but the line in question seems to
>> be this one:
>> 
>>this.loop = TransportCheck.createEventLoopGroup(config.
>> getInt(ExecConstants.BIT_SERVER_RPC_THREADS), "BitServer-“);
>> 
>> The config property here is defined in java-exec/src/main/resources
>> drill-module.conf:
>> 
>> drill.exec: {
>>  ...
>>  rpc: {
>>user: {
>>  ...
>>  server: {
>>   ...
>>threads: 1,
>> 
>> The only way you will get the error in your stack is if, somehow, your
>> build omits the drill-module.conf file. You can inspect your jar files to
>> see if this file is present.
>> 
>> Second, Eclipse works fine. The only trick is if you try to run certain
>> unit tests that use Mockito. “Modern” Eclipse Neon is based on JDK 8. But
>> the Mockito tests only run on JDK 7. There is no way to get the test runner
>> in Eclipse to use JDK 7. So, I end up building Drill for JDK 8 when using
>> Eclipse (just change the Java version in the root pom.xml file from 1.7 to
>> 1.8 — in two places.) Then run the Mockito-based tests outside of Eclipse,
>> after rebuilding back on JDK 7. Yes, a hassle, but this is just the way
>> that Drill works today.
>> 
>> Further, what works for me is:
>> 
>> 1. Check out Drill
>> 2. Change the pom.xml Java version number as noted above
>> 3. Build all of Drill without tests: “mvn clean install -DskipTests”
>> 4. Open Drill or refresh Drill in Eclipse. Eclipse does its own build.
>> 5. Run Drill directly from Eclipse.
>> 
>> Item 5 above is the third item. For many purposes, it is far more
>> convenient to run Drill directly from Eclipse. I use unit tests based on a
>> newer test framework. Find ExampleTest.java for example. For example, if a
>> particular query fails, and I can copy the data locally, then I just use
>> something like “fourthTest” to set up a storage plugin, set required
>> session, system and config options, run the query, and either display the
>> results or as summary. You can set breakpoints in the lines in question and
>> debug. Entire edit/compile/debug cycle is maybe 30 seconds vs. the five
>> minutes if you do a full external build.
>> 
>> I hope some of this helps you resolve your issue.
>> 
>> The most practical solution:
>> 
>> 1. Rebuild Drill
>> 2. Retry the run without the debugger.
>> 
>> If that works, review your Eclipse settings that might affect class path.
>> 
>> Thanks,
>> 
>> - Paul
>> 
>> 
>>> On Jul 13, 2017, at 8:20 AM, Muhammad Gelbana 
>> wrote:
>>> 
>>> To debug Drill in Eclipse, I ran *./drillbit.sh debug* and copied the 

Re: Debugging Drill in Eclipse

[Muhammad Gelbana]

When you said I'm rebuilding Drill, do you mean I'm running "*mvn clean
install*" after every change ? Actually I'm configuring eclipse to run the
same java command that "*drillbit.sh start*" command would run.

But I have to say that your method is much more efficient and practical. I
should've used test cases for development from the start.

I recently started exploring the test cases classes but I found it very
hard to comprehend so If there is some sort of a guide to the testing
classes architecture, packages structure\categorization..etc, it would be
very helpful if you share it. I looked into your github wiki
 but I didn't find
what I need.



- Gelbana

On Thu, Jul 13, 2017 at 6:53 PM, Paul Rogers  wrote:

> Hi Muhammad,
>
> There are several issues here.
>
> First, the problem you describe is not related to Eclipse. Second, several
> of us do use Eclipse and it works fine. Third, there are far easier ways to
> debug Drill in Eclipse then building Drill and doing remote debugging.
>
> First the error. The problem is that some bit of code is referring to a
> config parameter that does not appear in the config files. This kind of key
> would normally appear in some drill-module.conf file. (The file
> drill-override.conf is only for the very few config setting that you must
> modify for your site.)
>
> My source is a bit different than your, but the line in question seems to
> be this one:
>
> this.loop = TransportCheck.createEventLoopGroup(config.
> getInt(ExecConstants.BIT_SERVER_RPC_THREADS), "BitServer-“);
>
> The config property here is defined in java-exec/src/main/resources
> drill-module.conf:
>
> drill.exec: {
>   ...
>   rpc: {
> user: {
>   ...
>   server: {
>...
> threads: 1,
>
> The only way you will get the error in your stack is if, somehow, your
> build omits the drill-module.conf file. You can inspect your jar files to
> see if this file is present.
>
> Second, Eclipse works fine. The only trick is if you try to run certain
> unit tests that use Mockito. “Modern” Eclipse Neon is based on JDK 8. But
> the Mockito tests only run on JDK 7. There is no way to get the test runner
> in Eclipse to use JDK 7. So, I end up building Drill for JDK 8 when using
> Eclipse (just change the Java version in the root pom.xml file from 1.7 to
> 1.8 — in two places.) Then run the Mockito-based tests outside of Eclipse,
> after rebuilding back on JDK 7. Yes, a hassle, but this is just the way
> that Drill works today.
>
> Further, what works for me is:
>
> 1. Check out Drill
> 2. Change the pom.xml Java version number as noted above
> 3. Build all of Drill without tests: “mvn clean install -DskipTests”
> 4. Open Drill or refresh Drill in Eclipse. Eclipse does its own build.
> 5. Run Drill directly from Eclipse.
>
> Item 5 above is the third item. For many purposes, it is far more
> convenient to run Drill directly from Eclipse. I use unit tests based on a
> newer test framework. Find ExampleTest.java for example. For example, if a
> particular query fails, and I can copy the data locally, then I just use
> something like “fourthTest” to set up a storage plugin, set required
> session, system and config options, run the query, and either display the
> results or as summary. You can set breakpoints in the lines in question and
> debug. Entire edit/compile/debug cycle is maybe 30 seconds vs. the five
> minutes if you do a full external build.
>
> I hope some of this helps you resolve your issue.
>
> The most practical solution:
>
> 1. Rebuild Drill
> 2. Retry the run without the debugger.
>
> If that works, review your Eclipse settings that might affect class path.
>
> Thanks,
>
> - Paul
>
>
> > On Jul 13, 2017, at 8:20 AM, Muhammad Gelbana 
> wrote:
> >
> > To debug Drill in Eclipse, I ran *./drillbit.sh debug* and copied the VM
> > args and environment variables into a launcher. This worked fine for
> months.
> >
> > Obviously now I messed things up in a way and I can't debug Drill in
> > Eclipse anymore. I'm facing the following error:
> >
> > Exception in thread "main"
> > org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> > initializing values in Drillbit.
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:288)
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:272)
> > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:268)
> > Caused by: com.typesafe.config.ConfigException$Missing: *No
> configuration
> > setting found for key 'drill.exec.rpc'*
> > at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
> > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)
> > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
> > at 

Re: Debugging Drill in Eclipse

[Paul Rogers]

Hi Muhammad,

There are several issues here.

First, the problem you describe is not related to Eclipse. Second, several of 
us do use Eclipse and it works fine. Third, there are far easier ways to debug 
Drill in Eclipse then building Drill and doing remote debugging.

First the error. The problem is that some bit of code is referring to a config 
parameter that does not appear in the config files. This kind of key would 
normally appear in some drill-module.conf file. (The file drill-override.conf 
is only for the very few config setting that you must modify for your site.)

My source is a bit different than your, but the line in question seems to be 
this one:

this.loop = 
TransportCheck.createEventLoopGroup(config.getInt(ExecConstants.BIT_SERVER_RPC_THREADS),
 "BitServer-“);

The config property here is defined in java-exec/src/main/resources 
drill-module.conf:

drill.exec: {
  ...
  rpc: {
user: {
  ...
  server: {
   ...
threads: 1,

The only way you will get the error in your stack is if, somehow, your build 
omits the drill-module.conf file. You can inspect your jar files to see if this 
file is present.

Second, Eclipse works fine. The only trick is if you try to run certain unit 
tests that use Mockito. “Modern” Eclipse Neon is based on JDK 8. But the 
Mockito tests only run on JDK 7. There is no way to get the test runner in 
Eclipse to use JDK 7. So, I end up building Drill for JDK 8 when using Eclipse 
(just change the Java version in the root pom.xml file from 1.7 to 1.8 — in two 
places.) Then run the Mockito-based tests outside of Eclipse, after rebuilding 
back on JDK 7. Yes, a hassle, but this is just the way that Drill works today.

Further, what works for me is:

1. Check out Drill
2. Change the pom.xml Java version number as noted above
3. Build all of Drill without tests: “mvn clean install -DskipTests”
4. Open Drill or refresh Drill in Eclipse. Eclipse does its own build.
5. Run Drill directly from Eclipse.

Item 5 above is the third item. For many purposes, it is far more convenient to 
run Drill directly from Eclipse. I use unit tests based on a newer test 
framework. Find ExampleTest.java for example. For example, if a particular 
query fails, and I can copy the data locally, then I just use something like 
“fourthTest” to set up a storage plugin, set required session, system and 
config options, run the query, and either display the results or as summary. 
You can set breakpoints in the lines in question and debug. Entire 
edit/compile/debug cycle is maybe 30 seconds vs. the five minutes if you do a 
full external build.

I hope some of this helps you resolve your issue.

The most practical solution:

1. Rebuild Drill
2. Retry the run without the debugger.

If that works, review your Eclipse settings that might affect class path.

Thanks,

- Paul


> On Jul 13, 2017, at 8:20 AM, Muhammad Gelbana  wrote:
> 
> To debug Drill in Eclipse, I ran *./drillbit.sh debug* and copied the VM
> args and environment variables into a launcher. This worked fine for months.
> 
> Obviously now I messed things up in a way and I can't debug Drill in
> Eclipse anymore. I'm facing the following error:
> 
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:288)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:272)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:268)
> Caused by: com.typesafe.config.ConfigException$Missing: *No configuration
> setting found for key 'drill.exec.rpc'*
> at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
> at
> com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
> at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
> at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:98)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:1)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:55)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:286)
> ... 2 more
> 
> ​My *drill-override.conf*​ file has the following content (Aside from the
> comments)
> 
> drill.exec: {
>  cluster-id: "drillbits1",
>  zk.connect: "localhost:2181"
> }
> 
> I never needed to change this file to debug Drill in Eclipse !
> 
> Is there a standard way that Drill developers use to debug Drill ?
> 
> ​I appreciate any help for this because 

Debugging Drill in Eclipse

[Muhammad Gelbana]

To debug Drill in Eclipse, I ran *./drillbit.sh debug* and copied the VM
args and environment variables into a launcher. This worked fine for months.

Obviously now I messed things up in a way and I can't debug Drill in
Eclipse anymore. I'm facing the following error:

Exception in thread "main"
org.apache.drill.exec.exception.DrillbitStartupException: Failure while
initializing values in Drillbit.
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:288)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:272)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:268)
Caused by: com.typesafe.config.ConfigException$Missing: *No configuration
setting found for key 'drill.exec.rpc'*
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
at
com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:98)
at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:1)
at
org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:55)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:286)
... 2 more

​My *drill-override.conf*​ file has the following content (Aside from the
comments)

drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181"
}

I never needed to change this file to debug Drill in Eclipse !

Is there a standard way that Drill developers use to debug Drill ?

​I appreciate any help for this because it's totally blocking me !​