Glad to know the patch worked. It should be part of our 4.5.2 patch release which we hopefully will be releasing soon.
Out of curiosity - did your order by query ever finish? How much time did it take? Also how much time did the original query take with the patch that I provided? On Wed, Aug 26, 2015 at 6:02 AM, Sunil B <[email protected]> wrote: > Hi Samarth, > > The patch definitely solves the issue. The query "select * from > table" retrieves all the records. Thanks for the patch. > > Thanks, > Sunil > > On Tue, Aug 25, 2015 at 1:21 PM, Samarth Jain <[email protected]> > wrote: > > Thanks for the debugging info Sunil. > > > > I have uploaded a possible patch for the issue you are running into - > > https://issues.apache.org/jira/browse/PHOENIX-2207 > > > > Mind giving it a try? > > > > > > On Tue, Aug 25, 2015 at 11:18 AM, Sunil B <[email protected]> wrote: > >> > >> Hi Samarth, > >> > >> Answers to your questions: > >> 1) How many regions are there? > >> Ans: Total regions: 21. Each region is 5GB uncompressed (1.7GB > >> compressed). Total region servers: 7 > >> > >> 2) Do you have phoenix stats enabled? > >> http://phoenix.apache.org/update_statistics.html > >> Ans: The configuration is at default. In my understanding stats are > >> enabled by default. While debugging on the client I did notice that > >> phoenix client divides the query into ~1600 parallel scans using > >> guideposts. Let me know if this doesn't answer your question. > >> > >> 3) Is the table salted? > >> Ans: No > >> > >> 4) Do you have any overrides for scanner caching > >> (hbase.client.scanner.caching) or result size > >> (hbase.client.scanner.max.result.size) in your hbase-site.xml? > >> Ans: No. It is all at default configuration. > >> > >> > >> My Analysis: > >> ---------------- > >> As I said, it looks like a bug to me. Please let me know if you think > >> otherwise. > >> The chain of Iterators on the client side got from debugging: > >> RoundRobingResultIterator contains 1 ConcatResultIterator. The > >> ConcatResultIterator contains ~1600 LookAheadResultIterator. Each of > >> the LookAheadResultIterator contains one TableResultIterator > >> > >> In ParallelIterators.submitWork() function iterator.peek() is called > >> on each of the 1600 LookAheadResultIterators. This peek() function > >> retrieves data to cache from all the 1600 scanners. After this these > >> LookAheadResultIterators are added to ConcatResultIterator in > >> BaseResultIterators.getIterators() function. As ConcatResultIterator > >> goes serially through the rows, the purpose of peek() function is > >> lost. By the time ConcatResultIterator finishes with first couple of > >> LookAheadResultIterators, the scanners, which were initialized due to > >> peek(), timeout on the Region-Servers. > >> > >> > >> Workaround for now: > >> ---------------------------- > >> I am trying the query by adding an "order by" clause: > >> select * from TABLE order by PRIMARY_KEY > >> This modification seems to be working. It has been running for the > >> past 5 hours now. Will update the thread with success or failure. > >> Code Analysis: ScanPlan.newIterator function uses SerialIterators > >> instead of ParallelIterators if there is an "order by" in the query. > >> > >> Thanks, > >> Sunil > >> > >> On Mon, Aug 24, 2015 at 11:08 PM, Samarth Jain <[email protected]> > wrote: > >> > Sunil, > >> > > >> > Can you tells us a little bit more about the table - > >> > 1) How many regions are there? > >> > > >> > 2) Do you have phoenix stats enabled? > >> > http://phoenix.apache.org/update_statistics.html > >> > > >> > 3) Is the table salted? > >> > > >> > 4) Do you have any overrides for scanner caching > >> > (hbase.client.scanner.caching) or result size > >> > (hbase.client.scanner.max.result.size) in your hbase-site.xml? > >> > > >> > Thanks, > >> > Samarth > >> > > >> > > >> > On Mon, Aug 24, 2015 at 2:03 PM, Sunil B <[email protected]> wrote: > >> >> > >> >> Hi, > >> >> > >> >> Phoenix Version: 4.5.0-Hbase-1.0 > >> >> Client: slqline/JDBC driver > >> >> > >> >> I have a large table which has around 100GB of data. I am trying > to > >> >> execute a simple query "select * from TABLE", which times out with > >> >> scanner timeout exception. Please let me know if there is a way to > >> >> avoid this timeout without changing server side scanner timeout. > >> >> > >> >> Exception: WARN client.ScannerCallable: Ignore, probably already > >> >> closed > >> >> org.apache.hadoop.hbase.UnknownScannerException: > >> >> org.apache.hadoop.hbase.UnknownScannerException: Name: 15791, already > >> >> closed? > >> >> > >> >> > >> >> The reason for the timeout is that phoenix divides this query into > >> >> multiple parallel scans and executes scanner.next on each one of them > >> >> at the start of the query execution (this is because of the use of > >> >> PeekingResultIterator.peek function being called in submitWork > >> >> function of the ParallelIterators class). > >> >> > >> >> Is there a way I can force Phoenix to do a serial scan instead of > >> >> parallel scan with PeekingResultIterator? > >> >> > >> >> Thanks, > >> >> Sunil > >> > > >> > > > > > >
