[
https://issues.apache.org/jira/browse/TRAFODION-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640947#comment-14640947
]
Eric Owhadi commented on TRAFODION-1420:
----------------------------------------
previous discussion on dev list before JIRA creation for reference:
====================================================
what is codeGen, and do you know of a global setting to change the default?
Looking at source code I don't see where the default scan setting for small can
be anything but false? Also looking at the JNI API we have to trigger scan,
there is no param that I can see to play with the small param, so I don't see
how with this API stuff could be turned on or off?
Let me see about JIRA creation, that would be my first, so need to learn :-)
Eric
On Fri, Jul 24, 2015 at 1:31 PM, Qifan Chen <[email protected]> wrote:
During scan improvement work, I recall small scan is turned on by default by
hbase and should be turned off for large scans.
I wonder if we can confirm that first and how does the flag is set during
codeGen phase. This will make JIRA more accurate.
Secondly, welcome to the sql component and I am more than happy to provide any
help in compiler area.
Thanks
-Qifan
Sent from my iPhone
> On Jul 24, 2015, at 12:36 PM, Eric Owhadi <[email protected]> wrote:
>
> oh, I see,
> Eric
>
> On Fri, Jul 24, 2015 at 12:34 PM, Carol Pearson <[email protected]>
> wrote:
>
>> I saw that. I was more responding to your comment about metadata only being
>> read once, so not as necessary to optimize. That's a slightly different
>> tangent, but one that gets overlooked at times. Startup time to first
>> select is a key metric for a high-performance database, both for initial
>> install/setup/upgrade and on a simple restart.
>>
>> -Carol P.
>>
>> On Fri, Jul 24, 2015 at 10:30 AM, Eric Owhadi <[email protected]>
>> wrote:
>>
>>> Anoop is suggesting to use this not only for Meta data, but for any query
>>> where compiler evaluate that it would be appropriate to turn the feature
>>> on.
>>> Eric
>>>
>>> On Fri, Jul 24, 2015 at 12:26 PM, Carol Pearson <
>>> [email protected]>
>>> wrote:
>>>
>>>> If it only happens once, does that mean that this optimization might
>> be a
>>>> good one at startup time? If one of the failure modes is to bounce
>>>> something, or our users are simply restarting after some sort of
>>>> maintenance, startup would hit a lot of metadata all at once....
>>>>
>>>>
>>>> Thanks,
>>>> -Carol P.
>>>>
>>>> On Fri, Jul 24, 2015 at 9:57 AM, Eric Owhadi <[email protected]>
>>>> wrote:
>>>>
>>>>> more reading:
>>>>> https://issues.apache.org/jira/browse/HBASE-9488
>>>>>
>>>>> turns out that the performance improvement comes from 2 reasons: the
>> 3
>>>> RPC
>>>>> collapsed to 1, and the use of pread instead of seek+read.
>>>>> see facebook branch :
>> https://issues.apache.org/jira/browse/HBASE-7266
>>>>> they have enabled pread all the way, even for long scan using other
>>>>> features to implement prefetch needed for long scan.
>>>>>
>>>>> I incorrectly stated that the criteria was if your result set fit in
>>> your
>>>>> cache size. reading deeper, look like the criteria should be:
>>>>> if the scan range is within one datablock (64K) then we should set
>>> small
>>>>> scan.
>>>>>
>>>>> looking at the code, I seams that if you incorrectly set small on
>> non
>>>>> small, you just have incorrectly optimized. Will work slower...
>>>>> There were some bugs in early implementation not supporting well scan
>>>>> ranges crossing regions, but i see that the patch correcting them are
>>> in
>>>>> the branch of HBase we use.
>>>>>
>>>>> and yes the optimization works on the 2 cases you mention.
>>>>>
>>>>> on what I am observing, I only saw the traffic on first invoke, I
>> have
>>>> not
>>>>> tried to observe what happen on several run. So nothing to worry
>> about.
>>>>>
>>>>> Given the performance boost on small scan (3X), I think what you
>>> propose
>>>>> "We
>>>>> do have estimates of accessed rows at compile time and could turn
>> this
>>>> opt
>>>>> on, if rows are small." should be good candidate to add in the list
>> of
>>>>> stuff to do to improve perf...
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 24, 2015 at 9:29 AM, Anoop Sharma <
>> [email protected]>
>>>>> wrote:
>>>>>
>>>>>> There are 2 kind of scans that are done. One is a unique scan where
>>> we
>>>>> know
>>>>>> that only one unique row or a set of unique rows will be returned.
>>>>>> And second is a non-unique scan where multiple rows are returned.
>>>>>>
>>>>>> Does this optimization apply to both of these cases?
>>>>>>
>>>>>> We do have estimates of accessed rows at compile time and could
>> turn
>>>> this
>>>>>> opt on, if rows are small.
>>>>>> What happens if this flag is set and the scan is not small or
>> doesn't
>>>> fit
>>>>>> in
>>>>>> the cache? Will that work with some perf degradation or will it
>> fail?
>>>>>>
>>>>>> We only read metadata information from hbase when the table is used
>>> for
>>>>> the
>>>>>> first time
>>>>>> in a session or if table definition is changed. That info is then
>>>> cached
>>>>> in
>>>>>> compiler session structures.
>>>>>> Are you seeing metadata traffic every time you run a query from the
>>>> same
>>>>>> session? Are these queries on the same table?
>>>>>> Is table definition changing within your session?
>>>>>>
>>>>>> anoop
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Eric Owhadi [mailto:[email protected]]
>>>>>> Sent: Friday, July 24, 2015 6:46 AM
>>>>>> To: [email protected]
>>>>>> Subject: Small scanner: do you know about it, can it buy us perf
>>>>>> improvement?
>>>>>>
>>>>>> HI all,
>>>>>> stepping in the code, I have seen scan traffic on meta data. Now I
>>>> don't
>>>>>> know if this traffic is loading some cache, or if meta data is
>>> accessed
>>>>>> every time, no caching (apart from HBase block cache of course).
>>>>>> If so, I am not sure developers are aware of a new scanner
>> optimized
>>>> for
>>>>>> scans that result in result set fitting in the configured cache
>> size.
>>>> It
>>>>> is
>>>>>> 3 times faster than the regular scanner, and all you have to do is
>>> set
>>>> a
>>>>>> flag in the scan object to notify that it can benefit from small
>>>> scanner
>>>>>> perf optimization. In a nutshell, small scanner will do its job in
>>> one
>>>>> RPC
>>>>>> instead of 3 for regular scanner (combining the open/next/close).
>>> Small
>>>>>> scanner is already available in the Hbase .98 we use.
>>>>>>
>>>>>> Do you think we can take advantage of small scanner feature ether
>> for
>>>>> meta
>>>>>> data or somewhere else?
>>>>>>
>>>>>> regards,
>>>>>> Eric Owhadi
>>
> Use ClientSmallScanner for small scans to improve performance
> -------------------------------------------------------------
>
> Key: TRAFODION-1420
> URL: https://issues.apache.org/jira/browse/TRAFODION-1420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
> Reporter: Eric Owhadi
> Labels: performance
> Fix For: 2.0-incubating
>
>
> Hbase implements an optimization for small scan (defined as scanning less
> than a data block ie 64Kb) resulting in 3X performance improvement. The
> underlying trick is about cutting down RPC calls from 3 (OpenScan/Next/Close)
> to 1, and use pread stateless instead of seek/read state-full and locking
> method to read data. This JIRA is about improving the compiler who can be
> aware if a scan will be acting on single data block (small) or not, and pass
> this data to executor so that it can use the right parameter for scan.
> (scan.setSmall(boolean)).
> reference:
> https://issues.apache.org/jira/browse/HBASE-9488
> https://issues.apache.org/jira/browse/HBASE-7266
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
