altanh commented on pull request #7083: URL: https://github.com/apache/tvm/pull/7083#issuecomment-745532366
## Naming I propose we move everything PRNG to a new `random` namespace/module in Relay, so `relay.random.threefry_generate` etc. ## Handling different PRNG kernels The splitting, keygen, and bit-gen operations will be kernel-specific. However, AFAIK, most if not all of the commonly used random ops simply require random bits as inputs (i.e. don't care about how those bits were generated). I propose the following approach to handling different kernels (thanks to @jroesch for helpful discussion): - For each kernel `K` (e.g. `K = threefry`, `K = philox`, etc.), define `relay.random.K_key`, `relay.random.K_split` , and `relay.random.K_generate`. - In Python (and any other host language where we plan on writing Relay code directly), define `relay.random.key(seed: int, K: RandomAlg)` (where `RandomAlg` is some kind of enum with members `RandomAlg.THREEFRY` etc.) and `relay.random.split(key, K: RandomAlg)`. We can set a default K (for now obviously Threefry) to make it easier to use. - Now, for each random op `rop` that we care about, define `relay.random.rop_from_bits`, and in Python define `relay.random.rop(key, K)` which internally calls `relay.random.rop_from_bits(relay.random.K_generate(key))`. This should suffice to hide the algorithm from the user-facing API. **Problems.** - This might fail silently if the user inconsistently uses a key for kernel K with an op that uses kernel L ≠ K, if the keys are the same shape. This seems quite bad. Ideally, the keys will somehow be typed (perhaps we can use a Relay ADT?) but not sure how well supported this kind of use case is currently. ## Other notes - I will open a follow-up PR that adds some initial operators (currently I have a uniform random op), along with support for non-multiple of 4 sizes. - We should think about how to generalize over dtypes - currently, only `uint64` is supported for random bit generation in Threefry. Do we want to support more in the bit generation (is it possible?), or should we delegate this responsibility to auxiliary ops that split/join the `uint64`s as needed (e.g. converting 5 `uint64` to 10 `uint32`)? I think we need to come up with some kind of standardized approach for all the future random ops. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
