[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925401#comment-15925401
]
Yu Li commented on HBASE-17138:
-------------------------------
bq. It's almost impossible for us to fully upload our code base to github
(legal stuff you know...), and if we share the patches one by one, there'll be
70+ to generate and w/o git (cherry-pick/rebase) I'm afraid they're not easy to
apply.
[~tychang] [~anoop.hbase] Please allow me to quote my comment before. We have
separate commits before merging into our own master (squashed), but not
necessarily a one-one mapping between our commits and JIRAs. And you will need
to apply them manually, which I'm not sure whether is easier to resolve the
conflicts than using master version (where you could try cherry-pick). But yes,
if you think this could help, we could generate and upload the patches (which
requires some additional efforts than simply "git show", especially the mapping
between commits and JIRAs for a good reference, we did the work half a year
ago, so hopefully we could recall it correctly). Let me know your thoughts.
> Backport read-path offheap (HBASE-11425) to branch-1
> ----------------------------------------------------
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
> Reporter: Yu Li
> Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
> TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int
> llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport
> this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport this
> patch)
> HBASE-12296 Filters should work with ByteBufferedCell.
> HBASE-14120 ByteBufferUtils#compareTo small optimization.
> HBASE-13510 - Purge ByteBloomFilter (Ram)
> HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be
> easy to use in the CellComparators (Ram)
> HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot
> path (Ram)
> HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram)
> HBASE-13307 Making methods under ScannerV2#next inlineable, faster
> HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
> HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
> HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
> HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
> HBASE-12084 Remove deprecated APIs from Result.
> HBASE-12084 Remove deprecated APIs from Result - shell addendum
> HBASE-13754 Allow non KeyValue Cell types also to oswrite.
> HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
> HBASE-13817 ByteBufferOuputStream - add writeInt support.
> HBASE-12374 Change DBEs to work with new BB based cell.
> HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based
> reads
> HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk
> code.(Jingcheng)
> HBASE-13926 Close the scanner only after Call#setResponse.
> HBASE-12295 Prevent block eviction under us if reads are in progress from the
> BBs (Ram)
> HBASE-12295 - Addendum for multiGets to add the call back(Ram)
> HBASE-14001 Optimize write(OutputStream out, boolean withTags) for
> SizeCachedNoTagsKeyValue.
> HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and
> HBASE-12295 (ram)
> HBASE-14202 Reduce garbage we create.
> HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the
> Scan's start and stop Row (Ram)
> HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
> HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for
> ByteBufferedCells.
> HBASE-12298 Support BB usage in PrefixTree (Ram)
> HBASE-14398 - Create the fake keys required in the scan path to avoid copy to
> byte[] (Ram)
> HBASE-14480 Small optimization in SingleByteBuff.
> HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name
> (Anoop Sam John)
> HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
> HBASE-14832 Ensure write paths work with ByteBufferedCells in case of
> compaction (Ram)
> HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke)
> HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
> HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram)
> HBASE-12593 Tags to work with ByteBuffer.
> HBASE-15077 Support OffheapKV write in compaction with out copying data on
> heap.
> HBASE-14660 AssertionError found when using offheap BucketCache with
> assertion enabled (ram)
> HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner>
> additionalScanners) API into Region interface (Anoop Sam John
> HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
> HBASE-15785 Unnecessary lock in ByteBufferArray.
> HBASE-15760
> TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner
> fails in master branch (Ram)
> HBASE-15379 Fake cells created in read path not implementing
> SettableSequenceId
> HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not
> implementing SettableSequenceId
> HBASE-14940 Make our unsafe based ops more safe
> HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile
> Block served from L2 offheap cache.
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile
> Block served from L2 offheap cache - Addendum.
> HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
> HBASE-16704 Scan will be broken while working with DBE and
> KeyValueCodecWithTags
> the above jira ids are listed in chronological order I have backport，and
> there are also some patch i dont list, such as:
> 1. keep Cell api compatible with our existing code.
> 2. hfile format related compatibility issues.
> 3. client compatibility issue
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
