[GitHub] [incubator-livy] jerryshao commented on pull request #289: [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664907671 There still have some minor issues in JDBC side, we prepare a release when all the issues are done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-664909277 There still have some minor issues in JDBC side related to Scala versions. Will prepare a new release when everything is done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] giftkugel commented on pull request #289: [LIVY-756] Add Support Spark 3
giftkugel commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664894916 Is there any timeline to add this patch into a release? Maybe Livy 0.8? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao edited a comment on pull request #289: [LIVY-756] Add Support Spark 3
jerryshao edited a comment on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664907671 There still have some minor issues in JDBC side, will prepare a release when all the issues are done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] pdambrauskas commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
pdambrauskas commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-663497351 Hey, are there any plans to release this feature :) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jameschen1519 commented on pull request #212: [LIVY-11] Enable HA support
jameschen1519 commented on pull request #212: URL: https://github.com/apache/incubator-livy/pull/212#issuecomment-662927113 Taking over the pull request with @RogPodge 's permission. A bug involving query omission on failover was fixed, along with a bit of minor refactoring and renaming. I'm not too familiar with the code review structure of this repo, but can someone take a look at the PR and its changes? (Pinging @jerryshao based on prior messages as well) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] gmcoringa commented on pull request #249: [LIVY-702]: Submit Spark apps to Kubernetes
gmcoringa commented on pull request #249: URL: https://github.com/apache/incubator-livy/pull/249#issuecomment-658342256 Thanks @jahstreet for your effort. Tested with: * kubernetes 1.15.11 * Spark 2.4.5 What is holding this pr and https://github.com/apache/incubator-livy/pull/252? Also, @jahstreet if you could add me in the threads I can explain our use case, where we use jupyter to schedule spark jobs on kubernetes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] stczwd commented on pull request #289: [LIVY-756] Add Support Spark 3
stczwd commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-657356354 Glad to see this patch merged. I have test this path, it works well with spark3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-656291933 Thanks @groodt , I really appreciate your feedback. > Hopefully someday we can have a the ability to run interactive Spark jobs that autoscale on a Kubernetes cluster. We can already. Please refer the [Helm chart repo](https://github.com/jahstreet/spark-on-kubernetes-helm). > Does your branch support Spark 3.0? This branch is rebased to master, which already has the Spark 3.0 support merged #300 . In the meantime I'm preparing the update of Docker images and the corresponding aforementioned Helm charts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r452339853
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
Ok, I see that you need a global temp view:
```
scala> df.createGlobalTempView("aa")
Hive Session ID = fe4de24f-f7dc-496c-8dde-460c24f68548
scala> val spark2 = spark.newSession()
spark2: org.apache.spark.sql.SparkSession =
org.apache.spark.sql.SparkSession@59ede173
scala> spark2.sql("select * from global_temp.aa").show()
++
| id|
++
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
++
```
In any case, I still stand by my argument.
There is exactly zero chance that any actual users out there are doing this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r452339853
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
Ok, I see that you need a global temp view:
```
scala> df.createGlobalTempView("aa")
Hive Session ID = fe4de24f-f7dc-496c-8dde-460c24f68548
scala> val spark2 = spark.newSession()
spark2: org.apache.spark.sql.SparkSession =
org.apache.spark.sql.SparkSession@59ede173
scala> spark2.sql("select * from global_temp.aa").show()
++
| id|
++
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
++
```
In any case, I still stand by my argument.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] groodt commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
groodt commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-655930417 Thanks for your determination and efforts @jahstreet Hopefully someday we can have a the ability to run interactive Spark jobs that autoscale on a Kubernetes cluster. Does your branch support Spark 3.0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r451965507
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
My understanding of the implementation of the Livy Thrift server (from
ThriftSessionState.java) is that each Thrift session creates a new SparkSession
(with the same underlying SparkContext). So when you connect to the Thrift
server using the session id of an existing interactive scala session, the
Thrift session is using a new SparkSession, not the SparkSession in the
existing scala session.
```
scala> import org.apache.spark.sql.types.DecimalType
import org.apache.spark.sql.types.DecimalType
scala> val df = spark.range(10).select($"id".cast(DecimalType(10,-1)))
df: org.apache.spark.sql.DataFrame = [id: decimal(10,-1)]
scala> df.show()
++
| id|
++
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|0E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
|1E+1|
++
scala> df.createTempView("aa")
scala> val spark2 = spark.newSession()
spark2: org.apache.spark.sql.SparkSession =
org.apache.spark.sql.SparkSession@7abdfd36
scala> spark2.sql("select * from aa").show()
Hive Session ID = c73613e6-cb1e-4156-a6dc-31165d02dec0
org.apache.spark.sql.AnalysisException: Table or view not found: aa; line 1
pos 14
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:47)
...
Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException:
Table or view 'aa' not found in database 'default';
at
org.apache.spark.sql.hive.client.HiveClient$$anonfun$getTable$1.apply(HiveClient.scala:81)
...
```
I don't see how you can query the temp view from another SparkSession.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r451832688
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
I do not actually see the benefit. The fact that
org.apache.spark.sql.types.DecimalType allowed negative scale is actually a
mistake that Spark itself has now corrected. The SQL standard clearly states
that scale is a non-negative integer.
@jerryshao has already added support for Spark 3.0 to Livy. You yourself
said that if/when Spark no longer supports the case you're pushing for, the
test can be removed. Why add a test that should be removed? @jerryshao can you
comment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
mgaido91 commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r451698726
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
yes, the problem is Livy support Spark 2.4 currently... I can just
repeat what I said earlier: adding such a test can just bring a benefit and no
harm. If you think that adding this test can be harmful for Livy, please just
state why, otherwise I think we can add the test and go ahead with this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] liubarnabas commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
liubarnabas commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-654880389 This is a great feature! Hope the owner could merge it into master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet opened a new pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet opened a new pull request #167: URL: https://github.com/apache/incubator-livy/pull/167 ## What changes were proposed in this pull request? This PR is a new feature proposal: full support for Spark on Kubernetes (inspired by SparkYarnApp implementation). Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster: - Livy UI has cached logs/diagnostics page - Livy UI shows links to Spark UI and Spark History Server - With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI) - Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: https://github.com/kubernetes/ingress-nginx - Livy UI can be integrated with Grafana Loki logs (PR provides solution for that) Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 Design concept: https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md ## How was this patch tested? Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8: - Image: Spark 2.4.3 with Hadoop 3.2.0 (https://github.com/jahstreet/spark-on-kubernetes-docker) - History Server: https://github.com/helm/charts/tree/master/stable/spark-history-server - Jupyter Notebook with Sparkmagic: https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter What do you think on that? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-653879398 Squashed and rebased to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet closed pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet closed pull request #167: URL: https://github.com/apache/incubator-livy/pull/167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-653860581 Hi @JagadeeshNagella , the issue you've describe is cause by the incompatibility of Jackson libs.Fabric8 Java Kubernetes client `v4.9.2` is compiled with `jackson-core-2.10.3` (see [Maven Compile Dependencies](https://mvnrepository.com/artifact/io.fabric8/kubernetes-client/4.9.2)). Livy is bundled with `jackson-core-v2.9.9`, which has no `com/fasterxml/jackson/core/TSFBuilder` class. We need to upgrade Livy to `Jackson v2.10.3`. Once the version is upgraded it works well for both K8s API `v1.17.6` and `v1.18.1`. Please let me know if it fixes the issue for you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #258: Bump jackson.version from 2.9.9 to 2.10.1
jerryshao closed pull request #258: URL: https://github.com/apache/incubator-livy/pull/258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #243: [LIVY-693] Upgrade jackson to 2.9.10
jerryshao closed pull request #243: URL: https://github.com/apache/incubator-livy/pull/243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #243: [LIVY-693] Upgrade jackson to 2.9.10
jerryshao commented on pull request #243: URL: https://github.com/apache/incubator-livy/pull/243#issuecomment-652849861 Upgraded to 2.10.1, thanks a lot for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #258: Bump jackson.version from 2.9.9 to 2.10.1
jerryshao commented on pull request #258: URL: https://github.com/apache/incubator-livy/pull/258#issuecomment-652849346 Fixed in #300 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] dependabot[bot] commented on pull request #258: Bump jackson.version from 2.9.9 to 2.10.1
dependabot[bot] commented on pull request #258: URL: https://github.com/apache/incubator-livy/pull/258#issuecomment-652849356 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao edited a comment on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
jerryshao edited a comment on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-652848737 Fixed in #300 , thanks a lot for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
jerryshao closed pull request #299: URL: https://github.com/apache/incubator-livy/pull/299 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
jerryshao commented on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-652848737 Fixed in #300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-652848316 Thanks @tprelle for your work, I've already merged this PR, all credits to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #289: [LIVY-756] Add Support Spark 3
jerryshao closed pull request #289: URL: https://github.com/apache/incubator-livy/pull/289 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] dependabot[bot] commented on pull request #263: Bump jetty.version from 9.3.24.v20180605 to 9.4.24.v20191120
dependabot[bot] commented on pull request #263: URL: https://github.com/apache/incubator-livy/pull/263#issuecomment-652844566 Superseded by #301. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] dependabot[bot] opened a new pull request #301: Bump jetty.version from 9.3.24.v20180605 to 9.4.30.v20200611
dependabot[bot] opened a new pull request #301: URL: https://github.com/apache/incubator-livy/pull/301 Bumps `jetty.version` from 9.3.24.v20180605 to 9.4.30.v20200611. Updates `jetty-http` from 9.3.24.v20180605 to 9.4.30.v20200611 Commits https://github.com/eclipse/jetty.project/commit/271836e4c1f4612f12b7bb13ef5a92a927634b0d;>271836e Updating to version 9.4.30.v20200611 https://github.com/eclipse/jetty.project/commit/de6273baaeefcf8498221e85844a7286954e8727;>de6273b Merge pull request https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4959;>#4959 from eclipse/jetty-9.4.x-4903-fix-public-endpoint-check https://github.com/eclipse/jetty.project/commit/66ef0ebc03b77d6c11eabaf119bc4b3d3fc7242f;>66ef0eb Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - do ServerEndpointConfig validation check before started https://github.com/eclipse/jetty.project/commit/c48aee0a549e3233bacbd278d27e151b51bccf52;>c48aee0 maven war plugin 3.3.0 https://github.com/eclipse/jetty.project/commit/f41f601e1971691f4f4f27d6f9c9faf99749a4c3;>f41f601 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - fix validation on custom Configurator annotated endpoint https://github.com/eclipse/jetty.project/commit/b22e30679694b75234703e4b6d4f2abeb1704e6d;>b22e306 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - Improved behavior for Custom ServerEndpointConfig.Configurator https://github.com/eclipse/jetty.project/commit/cb09abe8737e17bd2afa4d68b3b5f2f163d53552;>cb09abe Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4920;>#4920 Restore ability to invalidate sessions on shutdown (https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4933;>#4933) https://github.com/eclipse/jetty.project/commit/9bfc168329462d60daca38a6984d277daac929f7;>9bfc168 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - Improved behavior for Custom ServerEndpointConfig.Configurator https://github.com/eclipse/jetty.project/commit/cbda92ab8c56b13bcf171877a16e5d2d6dc813c3;>cbda92a add olamy key https://github.com/eclipse/jetty.project/commit/1d16310bd25795f730e5362fc44753cb84c4764e;>1d16310 Merge pull request https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4953;>#4953 from lorban/jetty-9.4.x-4855-h2spec-8-1-2-6-failure Additional commits viewable in https://github.com/eclipse/jetty.project/compare/jetty-9.3.24.v20180605...jetty-9.4.30.v20200611;>compare view Updates `jetty-continuation` from 9.3.24.v20180605 to 9.4.30.v20200611 Updates `jetty-servlet` from 9.3.24.v20180605 to 9.4.30.v20200611 Commits https://github.com/eclipse/jetty.project/commit/271836e4c1f4612f12b7bb13ef5a92a927634b0d;>271836e Updating to version 9.4.30.v20200611 https://github.com/eclipse/jetty.project/commit/de6273baaeefcf8498221e85844a7286954e8727;>de6273b Merge pull request https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4959;>#4959 from eclipse/jetty-9.4.x-4903-fix-public-endpoint-check https://github.com/eclipse/jetty.project/commit/66ef0ebc03b77d6c11eabaf119bc4b3d3fc7242f;>66ef0eb Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - do ServerEndpointConfig validation check before started https://github.com/eclipse/jetty.project/commit/c48aee0a549e3233bacbd278d27e151b51bccf52;>c48aee0 maven war plugin 3.3.0 https://github.com/eclipse/jetty.project/commit/f41f601e1971691f4f4f27d6f9c9faf99749a4c3;>f41f601 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - fix validation on custom Configurator annotated endpoint https://github.com/eclipse/jetty.project/commit/b22e30679694b75234703e4b6d4f2abeb1704e6d;>b22e306 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - Improved behavior for Custom ServerEndpointConfig.Configurator https://github.com/eclipse/jetty.project/commit/cb09abe8737e17bd2afa4d68b3b5f2f163d53552;>cb09abe Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4920;>#4920 Restore ability to invalidate sessions on shutdown (https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4933;>#4933) https://github.com/eclipse/jetty.project/commit/9bfc168329462d60daca38a6984d277daac929f7;>9bfc168 Issue https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4903;>#4903 - Improved behavior for Custom ServerEndpointConfig.Configurator https://github.com/eclipse/jetty.project/commit/cbda92ab8c56b13bcf171877a16e5d2d6dc813c3;>cbda92a add olamy key https://github.com/eclipse/jetty.project/commit/1d16310bd25795f730e5362fc44753cb84c4764e;>1d16310 Merge pull request https://github-redirect.dependabot.com/eclipse/jetty.project/issues/4953;>#4953 from
[GitHub] [incubator-livy] dependabot[bot] closed pull request #263: Bump jetty.version from 9.3.24.v20180605 to 9.4.24.v20191120
dependabot[bot] closed pull request #263: URL: https://github.com/apache/incubator-livy/pull/263 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao closed pull request #300: URL: https://github.com/apache/incubator-livy/pull/300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652839568 I've triggered the travis test several times, seems everything is fine. I'm going to merge this, thanks @tprelle for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652320407 @tprelle @andrasbeni would you please help to review, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao closed pull request #300: URL: https://github.com/apache/incubator-livy/pull/300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652203533 reopen to trigger the test again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-651474672 Travis image seems too old to support Spark 3.0 R version, we should also upgrade Travis script. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on pull request #274: [LIVY-721][SERVER] Distributed session id generation
andrasbeni commented on pull request #274: URL: https://github.com/apache/incubator-livy/pull/274#issuecomment-651220914 @runzhiwang do you plan to continue working on this? If not, I am happy to address present concerns and move forward the feature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao commented on a change in pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#discussion_r447039881 ## File path: repl/src/main/scala/org/apache/livy/repl/Session.scala ## @@ -348,8 +348,10 @@ class Session( case "1" => (s"""setJobGroup(sc, "$jobGroup", "Job group for statement $jobGroup", FALSE)""", SparkR) - case "2" => + case "2" | "3" => Review comment: @tprelle this should be changed to support SparkR, otherwise it will be hung when running against spark 3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
tprelle commented on pull request #300: URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-651137090 @jerryshao How you manage to make sparkR work ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao opened a new pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao opened a new pull request #300: URL: https://github.com/apache/incubator-livy/pull/300 ## What changes were proposed in this pull request? This PR is based @tprelle 's PR #289 , and address all the left issues in that PR: 1. multi-scala version support in one build (Scala 2.11 and 2.12 support). 2. make SparkR work. Also reverts most of the unnecessary changes. Besides this PR remove the build below 2.4 (2.2, 2.3), since Spark 2.2 and 2.3 only ships with Scala 2.11, hard to maintain multiple version. But user could still use 2.2 and 2.3 without changes. All credits to @tprelle. ## How was this patch tested? Run UT and IT with Spark 2.4.5 and 3.0.0 locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446820287 ## File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala ## @@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T getJavaFutureResult(jobHandle, atMost) this } + + // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here Review comment: Yes, the comments seems confusing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
andrasbeni commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446670999 ## File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala ## @@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T getJavaFutureResult(jobHandle, atMost) this } + + // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here Review comment: In Scala 2.12 these two new methods ([see source](https://github.com/scala/scala/blob/v2.12.10/src/library/scala/concurrent/Future.scala#L248-L268 )) were added to the trait `Future`. As a result, `ScalaJobHandle`, which extends Future needs to implement them. At the same time I am not sure what @tprelle means by "the two subclasses below". AFAICT `ScalaJobHandle` does not have any subclasses. Also, I wouldn't say throwing an exception is a no-op. I suggest "They are implemented as unsupported operations here" if they are really not going to be [implemented](https://github.com/scala/scala/blob/v2.12.10/src/library/scala/concurrent/impl/Promise.scala#L31-L48). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446593365 ## File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala ## @@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T getJavaFutureResult(jobHandle, atMost) this } + + // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here Review comment: Would you please elaborate why the below two methods are needed for Scala 2.12? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r446464042
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
The message for the
[commit](https://github.com/apache/spark/commit/ff39c9271ca04951b045c5d9fca2128a82d50b46)
for [SPARK-30252](https://issues.apache.org/jira/browse/SPARK-30252) pretty
much says everything I pointed out, and concludes "it would be better to
disallow negative scale totally and make behaviors above be consistent."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r44643
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
The fact is that Spark SQL does not actually support a decimal type with
negative scale, and for good reason, the SQL standard itself states that scale
is a non-negative integer.
https://issues.apache.org/jira/browse/SPARK-30252 fixes
org.apache.spark.sql.types.DecimalType to disallow negative scale. The reason
given is the SQL standard.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
mgaido91 commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r446423157
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
I agree it is a corner case, but I think tests exist also for corner
cases, you don't want a software which behaves poorly even in corner cases.
Moreover I think it is not a big issue to just add a test for this case, which
if/when Spark won't support at all that case anymore can be removed. I see no
harm in adding a test and a benefit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] mmigdiso commented on pull request #289: [LIVY-756] Add Support Spark 3
mmigdiso commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-650323339 Hey @tprelle , will @jerryshao comments be a blocker for that PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] coheigea commented on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
coheigea commented on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-648887533 @tprelle It's up to the maintainers of the project, either it could all be done in one go (as in your PR), or maybe they might prefer to just update Jackson separately. If your PR is merged first, then I'll just close this PR. Either way it would be good from my POV to get us updated to Jackson 2.10.x ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
tprelle commented on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-648874686 Hi @coheigea I faced the same issue (shaded plugin and jackson version) on https://github.com/apache/incubator-livy/pull/289 for scala 2.12 and spark 3 support. I need also to change this. I do not know how deal with multiple PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444891736
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
I try to limit the number of version change, but because of scala 2.12
like @andrasbeni says I need to use scalatest 3.0.8 who are some breaking
change like org.scalatest.Suite =>org.scalatest.TestSuite, so to do not copy
the class I need to change also the global version of scalatest. So I need to
change the version of scalatra because of the incompatibility between scalatest
3.0.8 and scalatra-scalatest 2.3.0.
As I was working and easier to move forward with these upgrade of version I
choose to submit like this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648813898 > I just did a quick review about the overall implementation. Seems like we can only support one Scala version for one build here in this PR. But Livy could actually support different versions of Scala in one build (we used to support Scala 2.10 and 2.11 for one build). So I think we should have both Scala 2.12 and 2.11 support, and let Livy to choose jars automatically based on the Spark. I like the idea, I try it, but I was not able to make it work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444880828 ## File path: client-http/pom.xml ## @@ -109,58 +109,6 @@ - Review comment: It was a mistake, i will put it back This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444876495 ## File path: api/src/main/java/org/apache/livy/JobContext.java ## @@ -41,12 +41,12 @@ /** * @return The shared SQLContext instance. */ - SQLContext sqlctx(); + SQLContext sqlctx() throws Exception ; Review comment: It was before https://issues.apache.org/jira/browse/SPARK-31088, when they put back HiveContext class back to spark. When I revert the change I forgot to revert also the method signature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648718274 I'm doing some local improvements based on @tprelle 's current PR, I can submit a PR when everything is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444774680 ## File path: api/src/main/java/org/apache/livy/JobContext.java ## @@ -41,12 +41,12 @@ /** * @return The shared SQLContext instance. */ - SQLContext sqlctx(); + SQLContext sqlctx() throws Exception ; Review comment: Yes, I have the same concern about changing the signature of this interface. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy
Server's Scala version to 2.12, we could leave the Livy Server Scala upgrade to
another JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy
Server's Scala version to 2.12, we could leave the Livy Server Scala support to
another JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444717000
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in
3.0.0. So the version currently in use are incompatible with Spark 3.0's Scala
2.12.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444719602
##
File path: api/src/main/java/org/apache/livy/JobContext.java
##
@@ -41,12 +41,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
Review comment:
I believe adding throws clauses to these methods could break Java client
applications. I suggest to wrap exceptions that are not RuntimeException in a
RuntimeException instead.
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in
3.0.0. So the version currently in use are incompatibel with Spark 3.0's Scala
2.12.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444628517
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Are this artifacts upgrading necessary for Spark 3.0 support?
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Are these artifacts upgrading necessary for Spark 3.0 support?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444614548 ## File path: .travis.yml ## @@ -34,6 +34,12 @@ matrix: env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipITs' - name: "Spark 2.4 ITs" env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipTests' + - name: "Spark 3.0 Unit Tests" +env: MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipITs' + - name: "Spark 3.0 ITs" +env: + - MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipTests' + - PYSPARK_ROW_FIELD_SORTING_ENABLED=true Review comment: What is the purpose of setting this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444613917 ## File path: client-http/pom.xml ## @@ -109,58 +109,6 @@ - Review comment: What's the reason to remove this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648547180 I just did a quick review about the overall implementation. Seems like we can only support one Scala version for one build here in this PR. But Livy could actually support different versions of Scala in one build (we used to support Scala 2.10 and 2.11 for one build). So I think we should have both Scala 2.12 and 2.11 support, and let Livy to choose jars automatically based on the Spark. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
tprelle commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648289289 All unit and intregration scala and Python test are working. I'm currently redeploy it to make more integration test because of https://issues.apache.org/jira/browse/SPARK-29748 (the biggest change I spot for livy between 3.0.0-preview2 version en 3.0.0 version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] coheigea opened a new pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
coheigea opened a new pull request #299: URL: https://github.com/apache/incubator-livy/pull/299 In order to apply the PR to update Jackson to 2.10.1 (https://github.com/apache/incubator-livy/pull/258/files), it is also necessary to update the Maven Shade Plugin to avoid an ASM IllegalArgumentException error in the client-http module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] coheigea commented on pull request #243: [LIVY-693] Upgrade jackson to 2.9.10
coheigea commented on pull request #243: URL: https://github.com/apache/incubator-livy/pull/243#issuecomment-648264364 @peter-toth I think this PR and associated JIRA should be closed, as it is superceded by the upgrade to 2.10.1 (https://github.com/apache/incubator-livy/pull/258/files) which fixes many recent CVEs in Jackson Databind. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] peter-toth commented on pull request #244: [LIVY-694] Upgrade Jetty to 9.4.18.v20190429
peter-toth commented on pull request #244: URL: https://github.com/apache/incubator-livy/pull/244#issuecomment-648245292 @coheigea , unfortunately I didn't have time to check http mode, but we are using version 9.4.26.v20200117 now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] coheigea commented on pull request #244: [LIVY-694] Upgrade Jetty to 9.4.18.v20190429
coheigea commented on pull request #244: URL: https://github.com/apache/incubator-livy/pull/244#issuecomment-648213951 This PR duplicates an existing PR: https://github.com/apache/incubator-livy/pull/175/files @peter-toth is there any status on the Jetty check? It would be good to merge the fix due to the outstanding CVE issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648103728 Yes, I think it is fine to leave R support to another PR. Please make sure this PR is OK for Scala and Python part. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
tprelle commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648102517 @jerryshao the code it's working for spark scala and python but I do not managed to make it work for sparkR. I can submit the PR and open a ticket for sparkR support for spark3 if it's ok for the community This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648101189 I'm going to try this PR locally to see if the current code is complete or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella edited a comment on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella edited a comment on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-647843917 > > @JagadeeshNagella , I still need to have the Livy/Spark/Docker changes you introduced to have the same build: it can be diff patch, forked branch or whatever that can help me reproduce your problem. Could you please provide that? Otherwise it'll be hard to identify the exact problem, since [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) solves the compatibility issues you may have. > > @jahstreet , did you try to upgrade k8s client to 4.9.2? I sent you details in email. the issue still persist . anyone tried deploying this Livy patch in k8s 1.18.1 cluster, spark 2.4.5 and fabric k8s client 4.9.2 . ERROR: Exception in thread "kubernetesAppMonitorThread-org.apache.livy.utils.SparkKubernetesApp@776d88d" java.lang.NoClassDefFoundError: Could not initialize class io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-647843917 > > @JagadeeshNagella , I still need to have the Livy/Spark/Docker changes you introduced to have the same build: it can be diff patch, forked branch or whatever that can help me reproduce your problem. Could you please provide that? Otherwise it'll be hard to identify the exact problem, since [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) solves the compatibility issues you may have. > > @jahstreet , did you try to upgrade k8s client to 4.9.2? I sent you details in email. the issue still persist . anyone tried deploying this Livy patch in k8s 1.18.1 cluster, spark 2.4.5, fabric client 4.9.2 . ERROR: Exception in thread "kubernetesAppMonitorThread-org.apache.livy.utils.SparkKubernetesApp@776d88d" java.lang.NoClassDefFoundError: Could not initialize class io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r443859934
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
I think it is simply an accident that the CaseClass
`org.apache.spark.sql.types.DecimalType` allows a negative value for scale.
Other parts of the system enforce a check to disallow it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r443858197
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
That is an extremely contrived scenario - creating a temp view in an
interactive Livy session and connecting to it in a Livy Thrift session to query
the temp view. This is not a real use case.
Here is the SQL 99 standard on data types:
http://web.cecs.pdx.edu/~len/sql1999.pdf
In section `4.5 Numbers`, it says
```
4.5.1 Characteristics of numbers
An exact numeric value has a precision and a scale. The precision is a
positive integer that
determines the number of significant digits in a particular radix (binary or
decimal). The
scale is a non-negative integer. A scale of 0 (zero) indicates that the
number is an integer.
For a scale of S, the exact numeric value is the integer value of the
significant digits
multiplied by 10^-S.
```
The scale is a non-negative integer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r443858197
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
That is an extremely contrived scenario - creating a temp view in an
interactive Livy session and connecting to it in a Livy Thrift session to query
the temp view. This is not a real use case.
Here the SQL 99 standard on data types:
http://web.cecs.pdx.edu/~len/sql1999.pdf
In section `4.5 Numbers`, it says
```
4.5.1 Characteristics of numbers
An exact numeric value has a precision and a scale. The precision is a
positive integer that
determines the number of significant digits in a particular radix (binary or
decimal). The
scale is a non-negative integer. A scale of 0 (zero) indicates that the
number is an integer.
For a scale of S, the exact numeric value is the integer value of the
significant digits
multiplied by 10^-S.
```
The scale is a non-negative integer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-647246039 @tprelle is this PR ready for review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
mgaido91 commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r443259717
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
with Livy thriftserver, you can run SQL queries on an existing Spark
session. You can, for instance, create a Livy scala session, run the first two
commands of my example, then create a Livy thriftsession which re-uses the
scala session (by specifying its `sessionId` when connecting) and run the SQL
query from my example.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] wypoon commented on a change in pull request #296: [LIVY-771][THRIFT] Do not remove trailing zeros from decimal values.
wypoon commented on a change in pull request #296:
URL: https://github.com/apache/incubator-livy/pull/296#discussion_r443087452
##
File path:
thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/ResultSet.java
##
@@ -88,7 +88,7 @@ private String toHiveString(Object value, boolean
quoteStrings) {
} else if (quoteStrings && value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof BigDecimal) {
- return ((BigDecimal) value).stripTrailingZeros().toString();
+ return ((BigDecimal) value).toString();
Review comment:
What you have shown is that in Spark, you can create a `DataFrame` with
a negative scale `org.apache.spark.sql.types.DecimalType` column (because
`DecimalType` allows it), but I don't see how you can actually create a
persistent table with a column of decimal type with a negative scale.
Using your example, in spark-shell:
```
scala> val df = spark.range(10).select($"id".cast(DecimalType(10,-1)))
df: org.apache.spark.sql.DataFrame = [id: decimal(10,-1)]
scala> df.write.saveAsTable("neg_scale_dec_test")
Hive Session ID = e164a210-0268-4ab7-aca2-3c9a36892d54
[Stage 2:> (0 + 1)
/ 2]20/06/19 17:31:46 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 2.0
(TID 2, quasar-tnqtgl-2.vpc.cloudera.com, executor 2):
java.lang.IllegalArgumentException: Invalid DECIMAL scale: -1
at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:53)
at
org.apache.parquet.schema.Types$BasePrimitiveBuilder.decimalMetadata(Types.java:611)
at
org.apache.parquet.schema.Types$BasePrimitiveBuilder.build(Types.java:441)
at
org.apache.parquet.schema.Types$BasePrimitiveBuilder.build(Types.java:336)
at org.apache.parquet.schema.Types$Builder.named(Types.java:314)
at
org.apache.spark.sql.execution.datasources.parquet.SparkToParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:414)
...
scala> df.write.format("text").saveAsTable("neg_scale_dec_test")
20/06/19 17:35:49 ERROR command.CreateDataSourceTableAsSelectCommand: Failed
to write to table neg_scale_dec_test
org.apache.spark.sql.AnalysisException: Text data source does not support
decimal(10,-1) data type.;
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$$anonfun$verifySchema$1.apply(DataSourceUtils.scala:69)
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$$anonfun$verifySchema$1.apply(DataSourceUtils.scala:67)
...
scala> df.write.format("orc").saveAsTable("neg_scale_dec_test")
[Stage 3:> (0 + 2)
/ 2]20/06/19 17:36:25 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0
(TID 10, quasar-tnqtgl-4.vpc.cloudera.com, executor 4):
java.lang.IllegalArgumentException: Missing integer at
'struct<`id`:decimal(10,^-1)>'
at org.apache.orc.TypeDescription.parseInt(TypeDescription.java:244)
at org.apache.orc.TypeDescription.parseType(TypeDescription.java:362)
at org.apache.orc.TypeDescription.parseStruct(TypeDescription.java:333)
at org.apache.orc.TypeDescription.parseType(TypeDescription.java:391)
at org.apache.orc.TypeDescription.fromString(TypeDescription.java:412)
at
org.apache.spark.sql.execution.datasources.orc.OrcSerializer.org$apache$spark$sql$execution$datasources$orc$OrcSerializer$$createOrcValue(OrcSerializer.scala:216)
...
```
You can create a temp view from the DataFrame in Spark, but that won't be of
interest to users of Livy Thrift server. Users of the Thrift server want to run
SQL queries on tables created in Hive or Spark SQL.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
tprelle commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-646867581 > Spark 3 was released. https://spark.apache.org/downloads.html, please, continue Move to spark 3.0.0. Do not have time yet to test spark R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-646792497 > @JagadeeshNagella , I still need to have the Livy/Spark/Docker changes you introduced to have the same build: it can be diff patch, forked branch or whatever that can help me reproduce your problem. Could you please provide that? Otherwise it'll be hard to identify the exact problem, since [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) solves the compatibility issues you may have. @jahstreet , did you try to upgrade k8s client to 4.9.2? I sent you details in email. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] zjffdu commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
zjffdu commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r442677351
##
File path: assembly/pom.xml
##
@@ -29,7 +29,7 @@
pom
-apache-livy-${project.version}-bin
+
apache-livy-${project.version}-bin-${scala.binary.version}
Review comment:
Use `_${scala.binary.version}` to align with convention
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] zjffdu commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
zjffdu commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r442677351
##
File path: assembly/pom.xml
##
@@ -29,7 +29,7 @@
pom
-apache-livy-${project.version}-bin
+
apache-livy-${project.version}-bin-${scala.binary.version}
Review comment:
to `_${scala.binary.version}` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] Minutis commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Minutis commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-646481118 Spark 3 was released. https://spark.apache.org/downloads.html, please, continue :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] fifthpostulate commented on pull request #95: [LIVY-471][Server] Adding new API set to support resource uploading
fifthpostulate commented on pull request #95: URL: https://github.com/apache/incubator-livy/pull/95#issuecomment-646190343 This functionality would be extremely useful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645570904 @JagadeeshNagella , I still need to have the Livy/Spark/Docker changes you introduced to have the same build. Could you please share that? Otherwise it'll be hard to identify the exact problem, since [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) solves the compatibility issues you may have. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella commented on pull request #167:
URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645550398
> > could you please share with me the diff so that I could patch this PR
branch and try it out?
>
> @JagadeeshNagella , then the only I can do for you is to try it out on my
end. Also please tell me the version of your K8s cluster API.
@jahstreet ,
Thanks for your response . I will be waiting for reply then .
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1",
GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean",
BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6",
GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean",
BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc",
Platform:"linux/amd64"}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet edited a comment on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet edited a comment on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645532672 Hi @PedroRossi , please check the PR #249 , specifically to `livy.server.kubernetes.allowedNamespaces` and `livy.server.kubernetes.defaultNamespace` config options. The thing is this PR will not be supported anymore and all the work is splitted to smaller pieces. #249 and #252 are the first 2 PRs to go. Would really appreciate if you could help to move them forward to the merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645532672 Hi @PedroRossi , please check the PR #249 , specifically to `livy.server.kubernetes.allowedNamespaces` and `livy.server.kubernetes.defaultNamespace` config options. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] PedroRossi commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
PedroRossi commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645529277 Hi @jahstreet, are there any plans to support setting the namespace in which Livy is going to search for the pods? I would like to be able to set K8s Roles instead of ClusterRoles for Livy. Also thanks for the amazing job on enabling Spark K8s for Livy, I just applied this patch to the release 0.7.0 and it worked without a problem! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] codecov-commenter commented on pull request #298: [LIVY-775] Upgrade jQuery
codecov-commenter commented on pull request #298: URL: https://github.com/apache/incubator-livy/pull/298#issuecomment-645504498 # [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=h1) Report > Merging [#298](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/3b9bbefbaf8ddb3116952caf8c15b97d676386a7=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #298 +/- ## - Coverage 68.24% 68.19% -0.06% + Complexity 965 964 -1 Files 104 104 Lines 5952 5952 Branches900 900 - Hits 4062 4059 -3 - Misses 1311 1313 +2 - Partials579 580 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...in/scala/org/apache/livy/server/ui/UIServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/298/diff?src=pr=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvdWkvVUlTZXJ2bGV0LnNjYWxh) | `4.49% <0.00%> (ø)` | `1.00 <0.00> (ø)` | | | [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/298/diff?src=pr=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `50.00% <0.00%> (-2.95%)` | `7.00% <0.00%> (ø%)` | | | [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/298/diff?src=pr=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.00% <0.00%> (-0.84%)` | `45.00% <0.00%> (-1.00%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=footer). Last update [3b9bbef...1d1d2b9](https://codecov.io/gh/apache/incubator-livy/pull/298?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645473735 > could you please share with me the diff so that I could patch this PR branch and try it out? @JagadeeshNagella , then the only I can do for you is to try it out on my end. Also please tell me the version of your K8s cluster API. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni opened a new pull request #298: [LIVY-775] Upgrade jQuery
andrasbeni opened a new pull request #298: URL: https://github.com/apache/incubator-livy/pull/298 ## What changes were proposed in this pull request? The jquery version currently used by Livy is vulnerable (see [CVE](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-11022)) ## How was this patch tested? Verified manually This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella edited a comment on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella edited a comment on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645463429 > @JagadeeshNagella , could you please share with me the diff so that I could patch this PR branch and try it out? Also you might be interested in [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) which describes how to patch Spark to be deployed on the latest K8s API clusters. @jahstreet I did follow the above link to build spark image . we are able to run spark submit directly but not through Livy . I did use newly built spark image as base image for Livy however issue still persist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] JagadeeshNagella commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
JagadeeshNagella commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-645463429 > @JagadeeshNagella , could you please share with me the diff so that I could patch this PR branch and try it out? Also you might be interested in [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) which describes how to patch Spark to be deployed on the latest K8s API clusters. @jahstreet I did follow the above to build spark image . we are able to run spark submit directly but not through Livy . I did use newly built spark image as base image for Livy however issue still persist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #167: [LIVY-588]: Full support for Spark on Kubernetes
jahstreet commented on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-644922185 @JagadeeshNagella , could you please share with me the diff so that I could patch this PR branch and try it out? Also you might be interested in [this answer](https://stackoverflow.com/questions/60050782/spark-on-kubernetes-expected-http-101-response-but-was-403-forbidden/60052900#60052900) which describes how to patch Spark to be deployed on the latest K8s API clusters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jahstreet commented on pull request #249: [LIVY-702]: Submit Spark apps to Kubernetes
jahstreet commented on pull request #249: URL: https://github.com/apache/incubator-livy/pull/249#issuecomment-644914983 @lukatera , you can check that PR: https://github.com/apache/incubator-livy/pull/167 . Basically this PR is the clean backport from the former one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
