Ok, I'll rollback PHOENIX-1973, but are you saying this isn't the root cause of the pert regression?
On Friday, February 26, 2016, Sergey Soldatov <[email protected]> wrote: > James, > Well, its not, but I have an another problem with big loads when the > input is get split to several mappers. The optimization with replacing > table name in tablerowkeyPair cause problems with output name > generation, so the index instead of the table name appears in the > path. So lets just rollback the patch. > > Thanks, > Sergey > > On Fri, Feb 26, 2016 at 2:37 PM, James Taylor <[email protected] > <javascript:;>> wrote: > > Thanks, Sergey. Were you able to confirm whether or not PHOENIX-1973 was > > the root cause of the regression? > > > > On Fri, Feb 26, 2016 at 11:55 AM, Sergey Soldatov < > [email protected] <javascript:;>> > > wrote: > > > >> Oops. Small update. We can revert PHOENIX-1973 (bulk load > >> improvement), not PHOENIX-2649 (TableRowKeyPair comparator problem). > >> > >> On Fri, Feb 26, 2016 at 10:52 AM, Sergey Soldatov > >> <[email protected] <javascript:;>> wrote: > >> > Well, that's how MR bulk load works. Mappers gets all rows from the > >> > file and create the corresponding pairs <rowkey, column value>. MR > >> > engine sorts this stuff by rowkey and reducer sort it by value and put > >> > it to the hfile. After that HBase bulkload loads it into HBase. > >> > PHOENIX-2649 is just reduce the amount of data that is sending between > >> > mappers and reducer. Before that was N rows * K columns after it > >> > becomes N only. Because of the bug I mentioned before the phase when > >> > stuff is sorted by rowkey didn't work at all (first place why > >> > performance of 4.6 was better) and all values were written with a > >> > single rowkey and were received all at once by reducer (second place) > >> > and during the HBase bulkload there were no reason for splitting > >> > because of the same rowkey (third place). > >> > But of course we can reverse PHOENIX-2649 and see whether it helps. > >> > > >> > Thanks, > >> > Sergey > >> > > >> > On Fri, Feb 26, 2016 at 9:02 AM, 김영우 (Youngwoo Kim) < > [email protected] <javascript:;>> > >> wrote: > >> >> Exactly! Gabriel describes the fact that I observed. > >> >> > >> >> Many map and reduce tasks are launched but one or two tasks are > running > >> at > >> >> the end of the job. it looks like the work loads are skwed on > >> particular > >> >> task. > >> >> > >> >> Thanks, > >> >> Youngwoo > >> >> > >> >> 2016년 2월 26일 금요일, Gabriel Reid<[email protected] <javascript:;>>님이 > 작성한 메시지: > >> >> > >> >>> I just did a quick test run on this, and it looks to me like > something > >> >>> is definitely wrong. > >> >>> > >> >>> I ran a simple ingest test for a table with 5 regions, and it > appears > >> >>> that only a single HFile is being created. This HFile then needs to > be > >> >>> recursively split during the step of handing HFiles over to the > region > >> >>> servers (hence the "xxx no longer fits inside a single region. > >> >>> Splitting..." log messages). > >> >>> > >> >>> This implies that only a single reducer is actually doing any > >> >>> processing, which would certainly account for a performance > >> >>> degradation. My assumption is that the underlying issue is in the > >> >>> partitioner (or the data being passed to the partitioner). I don't > >> >>> know if this was introduced as part of PHOENIX-2649 or not. > >> >>> > >> >>> Sergey, are you (or someone else) able to take a look at this? > >> >>> Unfortunately, I don't think there's any way I can get a serious > look > >> >>> at this any more today. > >> >>> > >> >>> - Gabriel > >> >>> > >> >>> > >> >>> On Fri, Feb 26, 2016 at 11:21 AM, Sergey Soldatov > >> >>> <[email protected] <javascript:;> <javascript:;>> wrote: > >> >>> > I see. We will try to reproduce it. The degradation is possible > >> >>> > because 4.6 had a problem described in PHOENIX-2649. In two words > - > >> >>> > the comparator for rowkeys was working incorrectly and reported > that > >> >>> > all rowkeys are the same. If the input files are relatively small > and > >> >>> > reducer has enough memory, all records will be written in one step > >> >>> > with the same single rowkey. And that can be the reason why it was > >> >>> > faster and there were no splits. > >> >>> > > >> >>> > Thanks, > >> >>> > Sergey > >> >>> > > >> >>> > On Fri, Feb 26, 2016 at 1:37 AM, 김영우 (YoungWoo Kim) < > >> [email protected] <javascript:;> > >> >>> <javascript:;>> wrote: > >> >>> >> Sergey, > >> >>> >> > >> >>> >> I can't access the cluster right now, so I'll post details and > >> >>> >> configurations next week. important facts as far as I remember: > >> >>> >> - 8 nodes dev cluster (Hadoop 2.7.1, HBase 1.1.3, Phoenix 4.7.0 > RC2 > >> and > >> >>> >> Zookeeper 3.4.6) > >> >>> >> * 32 core / 256 GB RAM, Datanode/Nodemanager and RegionServer @ > >> same > >> >>> node, > >> >>> >> Assigned 24 GB for heap for region server > >> >>> >> - # of tables = 9 > >> >>> >> - Salted with 5, 10 or 20 buckets > >> >>> >> - Compressed using Snappy codec > >> >>> >> - Data Ingestion : 30 ~ 40 GB / day using bulk loading > >> >>> >> - Schema > >> >>> >> The table that I mentioned has 10 columns and 7 columns are > >> varchar > >> >>> and > >> >>> >> the rest are varchar[]. > >> >>> >> I can see performance degradation on bulk load from other > tables > >> >>> >> > >> >>> >> Thanks, > >> >>> >> > >> >>> >> Youngwoo > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> On Fri, Feb 26, 2016 at 6:02 PM, Sergey Soldatov < > >> >>> [email protected] <javascript:;> <javascript:;>> > >> >>> >> wrote: > >> >>> >> > >> >>> >>> Hi Youngwoo, > >> >>> >>> Could you provide a bit more information about the table > structure > >> >>> >>> (DDL would be great)? Do you have indexes? > >> >>> >>> > >> >>> >>> Thanks, > >> >>> >>> Sergey > >> >>> >>> > >> >>> >>> On Tue, Feb 23, 2016 at 10:18 PM, 김영우 (Youngwoo Kim) > >> >>> >>> <[email protected] <javascript:;> <javascript:;>> wrote: > >> >>> >>> > Gabriel, > >> >>> >>> > > >> >>> >>> > I'm using RC2. > >> >>> >>> > > >> >>> >>> > Youngwoo > >> >>> >>> > > >> >>> >>> > 2016년 2월 24일 수요일, Gabriel Reid<[email protected] > <javascript:;> > >> <javascript:;>>님이 > >> >>> 작성한 메시지: > >> >>> >>> > > >> >>> >>> >> Hi Youngwoo, > >> >>> >>> >> > >> >>> >>> >> Which RC are you using for this? RC-1 or RC-2? > >> >>> >>> >> > >> >>> >>> >> Thanks, > >> >>> >>> >> > >> >>> >>> >> Gabriel > >> >>> >>> >> > >> >>> >>> >> On Tue, Feb 23, 2016 at 11:30 AM, 김영우 (YoungWoo Kim) < > >> >>> [email protected] <javascript:;> <javascript:;> > >> >>> >>> >> <javascript:;>> wrote: > >> >>> >>> >> > Hi, > >> >>> >>> >> > > >> >>> >>> >> > I'm evaluating 4.7.0 RC on my dev cluster. Looks like it > works > >> >>> fine > >> >>> >>> but I > >> >>> >>> >> > run into performance degradation for MR based bulk loading. > >> I've > >> >>> been > >> >>> >>> >> > loading a million of rows per day into Phoenix table. From > >> 4.7.0 > >> >>> RC, > >> >>> >>> >> there > >> >>> >>> >> > are failed jobs with '600 sec' time out in map or reduce > >> stage. > >> >>> logs > >> >>> >>> as > >> >>> >>> >> > follows: > >> >>> >>> >> > > >> >>> >>> >> > 16/02/22 18:03:45 INFO mapreduce.Job: Task Id : > >> >>> >>> >> > attempt_1456035298774_0066_m_000002_0, Status : FAILED > >> >>> >>> >> > AttemptID:attempt_1456035298774_0066_m_000002_0 Timed out > >> after > >> >>> 600 > >> >>> >>> secs > >> >>> >>> >> > > >> >>> >>> >> > 16/02/22 18:05:14 INFO mapreduce.LoadIncrementalHFiles: > HFile > >> at > >> >>> >>> >> > > >> >>> >>> >> > >> >>> >>> > >> >>> > >> > hdfs://fcbig/tmp/74da7ab1-a8ac-4ba8-9d43-0b70f08f8602/HYNIX.BIG_TRACE_SUMMARY/0/_tmp/_tmp/f305427aa8304cf98355bf01c1edb5ce.top > >> >>> >>> >> > no longer fits inside a single region. Splitting... > >> >>> >>> >> > > >> >>> >>> >> > But, the logs have not seen before. so I'm facing about 5 ~ > >> 10x > >> >>> >>> >> performance > >> >>> >>> >> > degradation for bulk loading. (4.6.0: 10min but 60+ min > from > >> >>> 4.7.0 RC) > >> >>> >>> >> > furthermore, I can't find a clue from MR logs why the tasks > >> filed. > >> >>> >>> >> > > >> >>> >>> >> > And, I can see the hfile splitting after reduce stage. Is > it > >> >>> normal? > >> >>> >>> >> > > >> >>> >>> >> > My envs are: > >> >>> >>> >> > - Hadoop 2.7.1 > >> >>> >>> >> > - HBase 1.1.3 > >> >>> >>> >> > - Phoenix 4.7.0 RC > >> >>> >>> >> > > >> >>> >>> >> > Thanks, > >> >>> >>> >> > > >> >>> >>> >> > Youngwoo > >> >>> >>> >> > >> >>> >>> > >> >>> > >> >
