[
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175855#comment-13175855
]
Li Pi commented on HBASE-4608:
------------------------------
Okay. I'm confused.
I disabled compression, went back to trunk, and changed these lines of code in
HLogKey
System.out.println("Writing region: " + this.encodedRegionName.hashCode());
Bytes.writeByteArray(out, this.encodedRegionName);
System.out.println("Writing table: " + this.tablename.hashCode());
Bytes.writeByteArray(out, this.tablename);
And
in.readFully(this.encodedRegionName);
System.out.println("Reading region: " + this.encodedRegionName.hashCode());
this.tablename = Bytes.readByteArray(in);
System.out.println("Reading table: " + this.tablename.hashCode());
then I ran test replay after partial flush.
Got this as output
PositionWritten 124
Writing region: 1251181435
Writing table: 446506621
PositionWritten 319
Writing region: 1251181435
Writing table: 446506621
PositionWritten 514
Writing region: 1251181435
Writing table: 446506621
PositionWritten 709
Writing region: 1251181435
Writing table: 446506621
PositionWritten 904
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1099
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1294
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1489
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1684
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1879
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2074
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2289
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2484
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2679
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2874
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3069
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3264
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3459
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3654
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3849
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4044
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4239
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4454
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4649
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4844
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5039
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5234
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5429
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5624
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5819
Writing region: 1251181435
Writing table: 446506621
PositionWritten 124
Writing region: 736259394
Writing table: 510860944
PositionWritten 319
Writing region: 1336786910
Writing table: 403681456
PositionWritten 514
Writing region: 1336786910
Writing table: 403681456
PositionWritten 709
Writing region: 1336786910
Writing table: 403681456
PositionWritten 904
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1099
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1294
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1489
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1684
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1879
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2074
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2289
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2484
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2679
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2874
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3069
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3264
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3459
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3654
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3849
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4044
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4239
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4454
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4649
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4844
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5039
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5234
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5429
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5624
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5819
Writing region: 1336786910
Writing table: 403681456
PositionWritten 6014
Writing region: 1336786910
Writing table: 403681456
followed by
PositionRead 124
Reading region: 1037916733
Reading table: 256866950
PositionRead 319
Reading region: 720698180
Reading table: 966542180
PositionRead 514
Reading region: 1108113352
Reading table: 1082920280
PositionRead 709
Reading region: 717237635
Reading table: 787220834
PositionRead 904
Reading region: 173807871
Reading table: 611127977
PositionRead 1099
Reading region: 1961109485
Reading table: 788100239
PositionRead 1294
Reading region: 2069065824
Reading table: 586608097
PositionRead 1489
Reading region: 24862902
Reading table: 1258966396
PositionRead 1684
Reading region: 291843681
Reading table: 164096819
PositionRead 1879
Reading region: 606234185
Reading table: 1315525927
PositionRead 2074
Reading region: 1700109224
Reading table: 1465804433
PositionRead 2289
Reading region: 1990190694
Reading table: 2077192033
PositionRead 2484
Reading region: 1872332999
Reading table: 1222834702
PositionRead 2679
Reading region: 764334724
Reading table: 2074013561
PositionRead 2874
Reading region: 2138845270
Reading table: 843685757
PositionRead 3069
Reading region: 2139480405
Reading table: 780981467
PositionRead 3264
Reading region: 535465405
Reading table: 1610580905
PositionRead 3459
Reading region: 1899900
Reading table: 1866848242
PositionRead 3654
Reading region: 1382320624
Reading table: 1184634322
PositionRead 3849
Reading region: 828158517
Reading table: 1018679012
PositionRead 4044
Reading region: 1198520800
Reading table: 142476740
PositionRead 4239
Reading region: 162302775
Reading table: 518507735
PositionRead 4454
Reading region: 70862619
Reading table: 1282097095
PositionRead 4649
Reading region: 354961667
Reading table: 131165903
PositionRead 4844
Reading region: 1187109899
Reading table: 1632991863
PositionRead 5039
Reading region: 853232781
Reading table: 1535039248
PositionRead 5234
Reading region: 1683589725
Reading table: 847975203
PositionRead 5429
Reading region: 1217755329
Reading table: 1294658593
PositionRead 5624
Reading region: 1022661147
Reading table: 1554270688
PositionRead 5819
Reading region: 636371108
Reading table: 1020650096
PositionRead 6014
Reading region: 2114274883
Reading table: 206051672
PositionRead 6176
It doesn't seem like we're reading what we wrote.
> HLog Compression
> ----------------
>
> Key: HBASE-4608
> URL: https://issues.apache.org/jira/browse/HBASE-4608
> Project: HBase
> Issue Type: New Feature
> Reporter: Li Pi
> Assignee: Li Pi
> Attachments: 4608v1.txt
>
>
> The current bottleneck to HBase write speed is replicating the WAL appends
> across different datanodes. We can speed up this process by compressing the
> HLog. Current plan involves using a dictionary to compress table name, region
> id, cf name, and possibly other bits of repeated data. Also, HLog format may
> be changed in other ways to produce a smaller HLog.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
