Note: I did see after letting one just hang for a long time, a message about HEAP space... I was running with 12GB of Heap and 96 GB of Direct, I switched to 24 GB of Heap and 84 GB of Direct, and now my queries fail, but all with the index out of bounds issue, and then my drill bits stay responsive.
Could this hanging issue be related to Heap? I thought 12 GB would be quite a bit of Heap, but I guess not? Could we handle this better, provide better errors? Maybe the unresponsiveness is doing to some GC or other work? (Just spitballing ideas here). Thoughts? John On Mon, May 23, 2016 at 11:31 AM, John Omernik <[email protected]> wrote: > Hey all, this is separate, yet related issue to my other posts RE Parquet, > however, I thought I'd post this to see if this is normal or should be > handled (and/or JIRAed) > > I am running Drill 1.6, if you've read the other posts, I am trying to > CTAS a large amount of data (largish) 120 GB from Parquet to better > Parquet. > > As I am running, I sometimes get the Index Out of Bounds (as in the other > threads), but depending on source data and/or settings like using the new > parquet reader, I get a odd situation. > > When I refresh the profile in the WebUII get an error "VALIDATION ERROR: > no profile with given query id '' exists" > > I am running this in sqlline, and at this point, there is no error, but I > can't access my query profile. > > Other notes: > > 1. The webui is HORRIBLY slow > 2. If I cancel the query, it will show me some written parquet, but > obvious it wasn't finished > 3. There are no errors in any of the drillbits log files (except the > forman which starts to get "WARN" "Messos of mode (REQUEST OR RESPONSE) of > type 8 (or type 1) too longer than 500ms Actual duration was (high number > of ms betwen 1900 and 3500 ms) > 4. Like I said, no errors, just everything appears to hang. > > My memory and such seems good here, I have 96 GB of ram DIRECT per node, > and 12 GB of HEAP per node, 5 nodes,. > > The cluster seems really sluggish and out of sorts until I restart drill > bits... This seems like a very bad "error state" > > Has anyone seen this? Any thoughts on this? Should I open a JIRA? > > > Thanks, > John > > >
