Spark Kubernetes Architecture: Deployments vs Pods that create Pods

2019-01-29 Thread WILSON Frank
Hi, I've been playing around with Spark Kubernetes deployments over the past week and I'm curious to know why Spark deploys as a driver pod that creates more worker pods. I've read that it's normal to use Kubernetes Deployments to create a distributed service, so I am wondering why Spark just

Structured Streaming: stream-stream join with several equality conditions in a disjunction

2018-10-22 Thread WILSON Frank
Hi, I've just tried to do a stream-stream join with several equality conditions in a disjunction and got the following error: "terminated with exception: Stream stream joins without equality predicate is not supported;;" The query was in this sort of form: