Iterator parameters are passed as strings, so you have to encode binary data if you need to send that. The limit should be reasonable, but there's no hard-coded limit. If you are storing options for an iterator configured on a table, one would expect it to be able to be small enough to be stored easily in a ZooKeeper node, possibly alongside other options and table configuration, so it shouldn't be big enough to make ZooKeeper have trouble. If you are passing the option as part of a scan-time iterator, it should fit in an RPC message over Thrift.
I would keep them small (a few hundred characters or less), no more than a few thousand, if you really need to. If you need to pass large parameters, consider passing them indirectly, like passing the name of a file in HDFS that stores the binary data that the iterator reads when initialized. Your experience will vary depending on the configuration of your system's components and the hardware resources your machine has available. On Sun, May 2, 2021 at 8:39 PM Sanjay Deshmukh <sanj...@gmail.com> wrote: > > Is there a limit to how long an Iterator parameter can be? And does it have > to be a String, or is there a way to send arbitrary binary without encoding > it in a string? > > > -- > Sanjay Deshmukh > sanj...@gmail.com
