Typically, if you're using the OfflineScanner, you'd clone the table you
want to read and then take the clone offline. It's a simple (and fast)
solution that doesn't interrupt the availability of the table.
Doing the read offline will definitely be faster (maybe 20%, I'm not
entirely sure on accurate number and how it scales with nodes). The pain
would be the extra work in creating the clone, offline'ing the table,
and eventually deleting the clone when you're done with it. A little
more work, but manageable.
Ara Ebrahimi wrote:
Hi,
I’m trying to optimize a connector we’ve written for Presto. In some cases we
need to perform full table scans. This happens across all the nodes but each
node is assigned to process only a sharded subset of data. Each shard is hosted
by only 1 RFile. I’m looking at the AbstractInputFormat and OfflineIterator and
it seems like the code is not that hard to use for this case. Is there any
drawback? It seems like if the table is offline then OfflineIterator is used
which apparently reads the RFiles directly and doesn’t involve any RPC and I
think should be significantly faster. Is it so? Is there any drawback to using
this while the table is not offline but no other app is messing with the table?
Thanks,
Ara.
________________________________
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise confidential information. If you have received it in
error, please notify the sender immediately and delete the original. Any other
use of the e-mail by you is prohibited. Thank you in advance for your
cooperation.
________________________________
