Hi John,
The Drill scripts need quite a bit of TLC. (See DRILL-4581.) drill-config.sh
tries to set up both the Drillbit (server) and sqlline (client). Work was
needed to fully separate the two. The CLIENT_GC_OPTS are only for sqlline,
SERVER_GC_OPTS are for the drillbit.
The problem is that SERVER_GC_OPTS does two things that conflict. If it only
did logging, it would work:
$ loggc=/foo/bar.log
$ export SERVER_GC_OPTS="-Xloggc:<FILE-PATH>”
$ echo ${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}”}
-Xloggc:/foo/bar.log
But, current version of drill-env.sh helpfully adds other stuff to
SERVER_GC_OPTS, which makes the substitution fail:
export SERVER_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseG1GC "
Sigh… More bugs to fix… I’ve added this issue as a comment to DRILL-4581.
For now, just work around the problem using DRILL_JAVA_OPTS. The following
exists today in drill-env.sh:
export DRILL_JAVA_OPTS="-Xms$DRILL_HEAP -Xmx$DRILL_HEAP…
Add another line:
export DRILL_JAVA_OPTS=“$DRILL_JAVA_OPTS -Xloggc:/path/to/gc.log"
You’ll have to specify the log path, but it sounds like you do that anyway for
your Mesos setup.
By the way, another change we’re making for DoY is to split drill-env.sh into
three parts: Drill defaults move into drill-config.sh, distribution-specific
stuff moves into its own file, and drill-env.sh will contain only site-specific
settings.
- Paul
> On May 30, 2016, at 5:28 AM, John Omernik <[email protected]> wrote:
>
> More importantly, I am not sure how the strings inside the curly braces
> actually works either, based on testing... (echoing out the SERVER_GC_OPTS
> and CLIENT_GC_OPTS) It's not actually working
>
> If I am reading the bash correctly, than it's looking to, if SERVER_GC_OPTS
> (or CLIENT) is set (-n = return true if the length of the string is
> nonzero, since the Variable is interpreted, we are checking wether there is
> something in the variable) then we should be adding the xloggc (both of
> them) to the SERVER_GC_OPTS (and client).
>
> As you can see with the testing, the SERVER_GC_OPTS is only the value that
> I am setting from my drill-env.sh (default setting) which is loaded by
> drill-config.sh sourced earlier in the drillbit.sh. Thus, this code in
> drillbit.sh is effectively doing nothing ... I guess my thought process
> here would be to have someone help decide what is intended here, (I am not
> sure "Nothing" is intended based on the amount of code) and then we can do
> some updating here to clarify and ensure efficacy.
>
>
> Testing:
>
> if [ -n "$SERVER_GC_OPTS" ]; then
>
> export
> SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"}
>
> fi
>
> if [ -n "$CLIENT_GC_OPTS" ]; then
>
> export
> CLIENT_GC_OPTS=${CLIENT_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${loggc}"}
>
> fi
>
> echo "Server: $SERVER_GC_OPTS"
>
> echo "Client: $CLIENT_GC_OPTS"
>
> exit 1
>
>
> Server: -XX:+CMSClassUnloadingEnabled -XX:+UseG1GC
>
> Client:
>
> On Mon, May 30, 2016 at 6:43 AM, John Omernik <[email protected]> wrote:
>
>> So based on Paul's drilbit.sh comment and this, I decided to go ensure I
>> was enabling the proper GC logging because I am skipping the drillbit.sh.
>> I looked at the drillbit.sh, and frankly, It looks like a goofy error may
>> be in that... the <FILE-PATH> seems to be in documentation for other
>> hadoop-ish projects, but I don't think Java or BASH does anything with it.
>> Thus having that in the drillbit.sh (which to me shouldn't be changed)
>> seems to be a mistake... (Yes the -Xloggc after it may just overwrite what
>> was passed in the <FILE-PATH> but am I correct in saying that this is
>> actually just a mistake that in the drillbit.sh, and all it does is add
>> confusion? I hope I am wrong here and I get to learn something, but I ust
>> can't see how <FILE-PATH> is interpreted by bash or java....
>>
>>
>> John
>>
>>
>>
>> if [ -n "$SERVER_GC_OPTS" ]; then
>>
>> export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/
>> "-Xloggc:${loggc}"}
>>
>> fi
>>
>> if [ -n "$CLIENT_GC_OPTS" ]; then
>>
>> export CLIENT_GC_OPTS=${CLIENT_GC_OPTS/"-Xloggc:<FILE-PATH>"/
>> "-Xloggc:${loggc}"}
>>
>> fi
>>
>> On Mon, May 30, 2016 at 3:42 AM, Ted Dunning <[email protected]>
>> wrote:
>>
>>> On Sun, May 29, 2016 at 2:29 PM, John Omernik <[email protected]> wrote:
>>>
>>>> (It's a very weird situation that the bits get into,
>>>> everything hangs, somethings work, other things seem to be a in
>>>> an in-between between working and not working etc. Like describe table
>>>> operations eventually return but after 10+seconds. I resolve this by
>>>> restarting all bits, and then things are right as rain.
>>>>
>>>
>>> Sounds like GC pressure, possibly.
>>>
>>> The GC logging that was mentioned in connection with drill.sh would be
>>> helpful here.
>>>
>>
>>
