Hi, I have created a python UDF to make an API which requires an expirying OAuth token which requires refreshing every 600 seconds which is longer than any given stage.
Due to the nature of threads and local state, if I use a global variable, the variable goes out of scope regularly. I look into using a broadcast variable, but this doesn't support the ability to expire/refresh the variable. So I looked into using setLocalProperty and getLocalProperty on the spark context, but this can't be accessed within a UDF. Is there a recommended way to handle this scenario in PySpark? Thanks, Julian. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Handling-expirying-state-in-UDF-tp25021.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
