So I was looking through https://beam.apache.org/contribute/portability/ which lead me to BEAM-2900, and then to https://docs.google.com/document/d/1n6s3BOxOPct3uF4UgbbI9O9rpdiKWFH9R6mtVmR7xp0/edit# .
I was wondering if there is any considerations being given to native dependencies that user code may have (especially things like Python packages which can be super painful to deal with in a Spark cluster unless you use one of the vendor solutions)? Also, and this may be a terrible idea, but has there been thought given to the idea of a cross-language pipelines (I see these in Spark occasionally but with the DL stuff happening I suspect we might see users wanting cross-language functionality more often)? I also saw "Proposal: introduce an option to pass SDK harness container image in Beam SDKs" & it seems like Robert brought up the benefits of using Docker for Python runners, but I don't see the details on how we would expose that to users it in the design docs I've found yet (which could very well be I'm not looking at the right docs). Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau
