Yes, a reader works a batch at a time. Typically, we use 4095 as the default number of records to produce. I believe the Elastic approach is based on using cursormark.
-- Jacques Nadeau CTO and Co-Founder, Dremio On Tue, Sep 8, 2015 at 7:53 AM, Sudip Mukherjee <[email protected]> wrote: > Yes, there is no problem in that . Was just wondering if there is a > mechanism of getting data by batches from the record reader. > Actually , I wrote a basic plugin to solr and was thinking if I paginate > or use cursormark params in the recordreader. > > Thanks, > Sudip > > -----Original Message----- > From: Tomer Shiran [mailto:[email protected]] > Sent: 08 September 2015 PM 08:16 > To: [email protected] > Subject: Re: Drill Sql Max row size > > That won't be a problem. There's actually no limit on how many > records/rows you can have. > > > On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <[email protected]> > wrote: > > > > Hi, > > I have somewhere around a million records. But columns are less in > > numbers (up to 10) > > > > Thanks, > > Sudip > > > > -----Original Message----- > > From: Jacques Nadeau [mailto:[email protected]] > > Sent: 08 September 2015 AM 06:58 > > To: [email protected] > > Subject: Re: Drill Sql Max row size > > > > Generally, no. That being said, Drill will probably struggle if you > start reading records where one or more cells is greater than a few hundred > kilobytes (or mbs) or more than several hundred columns/fields. What size > records are you working with? > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee > > <[email protected]> > > wrote: > > > >> Hi Devs, > >> > >> Is there a max row limit which I can configure while pulling data > >> from underlying datasource? If there is a large data-set would drill > >> fetch like page by page? > >> > >> Thanks, > >> Sudip > >> > >> > >> > >> ***************************Legal > >> Disclaimer*************************** > >> "This communication may contain confidential and privileged material > >> for the sole use of the intended recipient. Any unauthorized review, > >> use or distribution by others is strictly prohibited. If you have > >> received the message by mistake, please advise the sender by reply > >> email and delete the message. Thank you." > >> ********************************************************************* > >> * > > > > > > > > ***************************Legal Disclaimer*************************** > > "This communication may contain confidential and privileged material > > for the sole use of the intended recipient. Any unauthorized review, > > use or distribution by others is strictly prohibited. If you have > > received the message by mistake, please advise the sender by reply email > and delete the message. Thank you." > > ********************************************************************** > > > > ***************************Legal Disclaimer*************************** > "This communication may contain confidential and privileged material for > the > sole use of the intended recipient. Any unauthorized review, use or > distribution > by others is strictly prohibited. If you have received the message by > mistake, > please advise the sender by reply email and delete the message. Thank you." > ********************************************************************** >
