Hi Charles, The Varchar column can hold any length of data. We’ve recently been working on tests that have columns up to 8K in length.
The one caveat is that, when working with data larger than 256 bytes, you must be extremely careful in your reader. The out-of-box text reader will always read 64K rows. This (due to various issues) can cause memory fragmentation and OOM errors when used with columns greater than 256 bytes in width. If you are developing your own storage plugin, then adjust the size of each row batch so that no single vector is larger than 16 MB in size. Then you can use any size of column. Suppose your logs contain text lines up to, say, 1K in size. This means that each record batch your reader produces must be of size less than 16 MB / 1K / row = 1600 rows (rather than the usual 64K.) Once the data is in the Varchar column, the rest of Drill should “just work” on that data. - Paul > On Jan 26, 2017, at 4:11 PM, Charles Givre <[email protected]> wrote: > > I’m working on a plugin to read log files and the data has some long strings. > Is there a data type that can hold strings longer than 256 characters? > Thanks, > — Charles
