[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[suryag10]

Github user suryag10 commented on the issue:

https://github.com/apache/spark/pull/20272
  
> If this can get a rebase, and maybe a few sentences in the k8s docs, I'll 
merge.

Hi, I had rebased this patch to the latest master and this patch was 
missing another fix to run the STS on K8S cluster mode. Following is the PR

 https://github.com/apache/spark/pull/22433

Can you please review this once?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[deveshk0]

Github user deveshk0 commented on the issue:

https://github.com/apache/spark/pull/20272
  
is this coming with spark 2.4 ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[deveshk0]

Github user deveshk0 commented on the issue:

https://github.com/apache/spark/pull/20272
  
I have built spark with the same changes for thrift server. It is running 
fine for me.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/20272
  
still need to run tests 
https://github.com/apache/spark/pull/20272#pullrequestreview-108271893



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/20272
  
If this can get a rebase, and maybe a few sentences in the k8s docs, I'll 
merge.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[liyinan926]

Github user liyinan926 commented on the issue:

https://github.com/apache/spark/pull/20272
  
@felixcheung I think yes and with 
https://github.com/apache/spark/pull/21748, users should be able to run the 
Thrift server in a pod.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/20272
  
Is this aligned with the "in cluster client"? @foxish @mccheah 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20272
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[nrchakradhar]

Github user nrchakradhar commented on the issue:

https://github.com/apache/spark/pull/20272
  
Its working for us in our test environment. No issues so far. Only thing to 
take care is the clean-up of the driver manually when the thrift server is 
re-installed or started due to any reason.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[ah-]

Github user ah- commented on the issue:

https://github.com/apache/spark/pull/20272
  
Is anyone still looking at this? It seems like it should just work?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/20272
  
It's really not that much work to write a couple of sentences about this in 
the running-on-k8s docs, even if you're just pointing people to the kubernetes 
documentation. Is that really so controversial?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[ozzieba]

Github user ozzieba commented on the issue:

https://github.com/apache/spark/pull/20272
  
Ultimately I see this as being in the realm of Kubernetes knowledge, rather 
than Spark knowledge. These features are well-documented in Kubernetes 
documentation, and I am not sure there's a need to replicate that here. For a 
user new to both Spark and Kubernetes a combined tutorial could certainly be 
helpful, but that should likely be a more comprehensive undertaking.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/20272
  
Great, how about explaining that in some place more visible to Spark users 
than a PR on github.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[ozzieba]

Github user ozzieba commented on the issue:

https://github.com/apache/spark/pull/20272
  
@vanzin In general Kubernetes makes this super easy:
- The most basic workflow is to use the driver pod name (which is in the 
output of Spark Submit, or can be found with `kubectl get pods`), and run  
`kubectl port-forward spark-app-driver-podname 31416:1`, which will 
automatically 
[forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
 localhost:31416 to the pod's port 1. Any jdbc application can then be used 
to query `jdbc:hive2://localhost:31416`. This is the approach I took in the 
integration tests linked above.
- Alternatively, any other application on the cluster can simply use 
spark-app-driver-podname:1, which will be resolved by 
[kube-dns](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/).
- For persistent external access one can run `kubectl expose pod 
spark-app-driver-podname --type=NodePort --port 1` to create a Kubernetes 
[Service](https://kubernetes.io/docs/concepts/services-networking/service/) 
which will accept connections on a particular port of *every* node on the 
cluster and send them to the driver's port 1. 
- On a cloud environment, using `type=LoadBalancer` in the above will 
create a global [load 
balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/)
 that can be used to access Thrift from across the Internet.
- One can also define a service that automatically selects the driver using 
a user-specified label (eg `bin/spark-submit.sh --conf 
spark.kubernetes.driver.label.mythriftserver=true` followed by `kubectl expose 
pod --selector=mythriftserver=true`)

Perhaps the key point is that these are all core Kubernetes features, and 
not specific to Spark in any way. Users familiar with Kubernetes should be able 
to find the approach that works best for their environment and use case.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/20272
  
So how are you expected to contact the thrift server after it's up, and 
even figure out where it started? Isn't the container unreachable from the 
outside world unless you do some port mapping on the host?

I see part of that is explained in the bug, but sounds like this change 
needs better user documentation.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/20272
  
>IIUC there was a issue in launching Thrift Server on YARN cluster mode, 
and I'm not sure whether it has been fixed (maybe @jerryshao can kindly check 
that?)

Sorry I cannot remember the issue. Yarn cluster mode doesn't support 
thriftserver.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/20272
  
Makes sense. The change LGTM.

On Jan 29, 2018 10:23 AM, "Jiang Xingbo"  wrote:

> IIUC there was a issue in launching Thrift Server on YARN cluster mode,
> and I'm not sure whether it has been fixed (maybe @jerryshao
>  can kindly check that?) Anyway that is not
> a problem on Spark side, therefore should not affect the Kubernetes 
cluster
> mode.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/20272
  
IIUC there was a issue in launching Thrift Server on YARN cluster mode, and 
I'm not sure whether it has been fixed (maybe @jerryshao can kindly check 
that?) Anyway that is not a problem on Spark side, therefore should not affect 
the Kubernetes cluster mode.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[ozzieba]

Github user ozzieba commented on the issue:

https://github.com/apache/spark/pull/20272
  
@andrewor14 @liancheng can you chime in?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[ozzieba]

Github user ozzieba commented on the issue:

https://github.com/apache/spark/pull/20272
  
@foxish per SPARK-5176 and the associated 
[PR](https://github.com/apache/spark/pull/4137), it seems there was a technical 
issue with spark-internal that wouldn't allow Thrift Server to run on YARN 
cluster mode. It was deemed a minor fix at the time, so perhaps the changes 
were made in the meantime. Regardless, it does now work at least on Kubernetes


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/20272
  
@jiangxb1987 Is there any specific owner of the thrift server that we can 
ping here? The testing looks good - so, all we're waiting for is confirmation 
from them on the original intent behind disallowing the thrift server in 
cluster mode.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/20272
  
cc @liyinan926 Do you have some time to verify this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org