hhvu0102 opened a new issue, #8683: URL: https://github.com/apache/texera/issues/8683
### Feature Summary Opening a new thread since I can't reopen the old discussion here https://github.com/apache/texera/issues/4074 My question is: how can I write a function in a 2-port UDF to access input data at the same time? For example, if data from `port 0` is `df0`, and data from `port 1` is `df1`, how should I design my code so a function `f(df0, df1)` could be executed properly? Attached here is an example in my use case, where `port 0` is a df of 19.5k lines and ~200 columns (say `df0`, while data from `port 1` is smaller with 63 lines (say `df1`). AI bot suggested that we put a parameter with a fix number of lines matching with n lines in `df0` to check if `df0` has finished loading, which in general cases would not work. This is other things I have tried (in pseudocode): 1) Attempt 1: ``` if port == 0: df0 <- df0 else if port == 1: df1 <- df1 #a function that does something with both df0 and df1 function_to_called(df0, df1) ``` This will not work because `df1` will finish loading before `df0`, and `function_to_called` will throw an error that `df0` is not found. 2) Attempt 2: Same logic as above, but I forced `port 1` data to depend on `port 0` so `port 0` finishes first (as recommended in https://github.com/apache/texera/issues/4074). Now `function_to_called` will throw an error that `df1` is not found, because it's called before `df1` is loaded. The attached screenshot is in R, but the question stands the same for Python-based UDF. <img width="2766" height="1738" alt="Image" src="https://github.com/user-attachments/assets/641fdfa6-698c-4d0f-8ffc-75ea613f9c38"; /> ### Proposed Solution or Design I expect a way to check if data has finished loading. ### Affected Area _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
