Yep, I will create all clean logs tomorrow run the query that caused it. Thanks. John
On Monday, June 13, 2016, Parth Chandra <[email protected]> wrote: > Yes, we can discuss this on the hangout. > You're right, there are two issues - > Limiting memory usage to a maximum limit should be the goal of every > component. We are not there yet with Drill though. > Getting an Out of Memory Error and having the Drillbit become > unresponsive is something we should rarely see as either the Drill > allocator or the JVM successfully catch the condition. Can you grep your > logs so we can see if that indeed is what happened? > > > > On Mon, Jun 13, 2016 at 4:27 PM, John Omernik <[email protected] > <javascript:;>> wrote: > > > I'd like to talk about that on the hangout. Drill should do better at > > failing with a clean oom error rather then having a bit go unresponsive. > > Can just that bit be restarted to return to a copacetic state? As an > admin, > > if this is the case, how do I find this bit? > > > > Other than adding RAM, are there any query tuning settings that could > help > > prevent the unresponsive bit? ( I see this as two issues, the memory > > settings for the 1024m block size CTAS and the how can we prevent a bit > > from going unresponsive? ) > > On Jun 13, 2016 6:19 PM, "Parth Chandra" <[email protected] > <javascript:;>> wrote: > > > > The only time I've seen a drillbit get unresponsive is when you run out > of > > Direct memory. Did you see any 'Out of Memory Error' in your logs? If you > > see those then you need to increase the Direct memory setting for the > JVM. > > ( DRILL_MAX_DIRECT_MEMORY in drill-env.sh) > > > > > > > > > > On Mon, Jun 13, 2016 at 4:10 PM, John Omernik <[email protected] > <javascript:;>> wrote: > > > > > The 512m block size worked. My issue with the 1024m block size was on > > the > > > writing using a CTAS.... that's where my nodes got into a bad > > state....thus > > > I am wondering what setting on drill would be the right setting to help > > > node memory pressures on a CTAs using 1024m block size > > > On Jun 13, 2016 6:06 PM, "Parth Chandra" <[email protected] > <javascript:;>> wrote: > > > > > > In general, you want to make the Parquet block size and the HDFS block > > size > > > the same. A Parquet block size that is larger than the HDFS block size > > can > > > split a Parquet block ( i.e. row_group ) across nodes and that will > > > severely affect performance as data reads will no longer be local. 512 > MB > > > is a pretty good setting. > > > > > > Note that you need to ensure the Parquet block size in the source file > > > which (maybe) was produced outside of Drill. So you will need to make > the > > > change in the application used to write the Parquet file. > > > > > > If you're using Drill to write the source file as well then, of course, > > the > > > block size setting will be used by the writer. > > > > > > If you're using the new reader, then there is really no knob you have > to > > > tweak. Is parquet-tools able to read the file(s)? > > > > > > > > > > > > On Mon, Jun 13, 2016 at 1:59 PM, John Omernik <[email protected] > <javascript:;>> wrote: > > > > > > > I am doing some performance testing, and per the Impala > documentation, > > I > > > am > > > > trying to use a block size of 1024m in both Drill and MapR FS. When > I > > > set > > > > the MFS block size to 512 and the Drill (default) block size I saw > some > > > > performance improvements, and wanted to try the 1024 to see how it > > > worked, > > > > however, my query hung and I got into that "bad state" where the > nodes > > > are > > > > not responding right and I have to restart my whole cluster (This > > really > > > > bothers me that a query can make the cluster be unresponsive) > > > > > > > > That said, what memory settings can I tweak to help the query work. > > This > > > is > > > > quite a bit of data, a CTAS from Parquet to Parquet, 100-130G of data > > per > > > > data (I am doing a day at a time), 103 columns. I have to use the > > > > "use_new_reader" option due to my other issues, but other than that I > > am > > > > just setting the block size on MFS and then updating the block size > in > > > > Drill, and it's dying. Since this is a simple CTAS (no sort) which > > > settings > > > > can be beneficial for what is happening here? > > > > > > > > Thanks > > > > > > > > John > > > > > > > > > > -- Sent from my iThing
