On Sat, Nov 18, 2017 at 10:33 PM, Holden Karau <[email protected]> wrote:
> So I was looking through https://beam.apache.org/contribute/portability/ > which lead me to BEAM-2900, and then to > https://docs.google.com/document/d/1n6s3BOxOPct3uF4UgbbI9O9rpdiKW > FH9R6mtVmR7xp0/edit# > . > > I was wondering if there is any considerations being given to native > dependencies that user code may have (especially things like Python > packages which can be super painful to deal with in a Spark cluster unless > you use one of the vendor solutions)? > Yes. Users should eventually be able to customize the container, and put these dependencies in their container image. This has been a common user request actually. > Also, and this may be a terrible idea, but has there been thought given to > the idea of a cross-language pipelines (I see these in Spark occasionally > but with the DL stuff happening I suspect we might see users wanting > cross-language functionality more often)? > Yes, this is an explicit goal. > > I also saw "Proposal: introduce an option to pass SDK harness container > image in Beam SDKs" & it seems like Robert brought up the benefits of using > Docker for Python runners, but I don't see the details on how we would > expose that to users it in the design docs I've found yet (which could very > well be I'm not looking at the right docs). > > Cheers, > > Holden :) > > -- > Twitter: https://twitter.com/holdenkarau >
