[GitHub] felixcheung commented on a change in pull request #3307: ZEPPELIN-4005. SparkRInterpreter is broken for spark 2.1.3 and 2.2.2
felixcheung commented on a change in pull request #3307: ZEPPELIN-4005.
SparkRInterpreter is broken for spark 2.1.3 and 2.2.2
URL: https://github.com/apache/zeppelin/pull/3307#discussion_r257450587
##
File path:
spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
##
@@ -86,7 +86,11 @@ public boolean isSpark2() {
}
public boolean isSecretSocketSupported() {
-return this.newerThanEquals(SPARK_2_3_1);
+return this.newerThanEquals(SPARK_2_3_1) ||
+(this.newerThanEquals(SparkVersion.fromVersionString("2.2.2")) &&
+this.olderThan(SparkVersion.fromVersionString("2.3.0"))) ||
+(this.newerThanEquals(SparkVersion.fromVersionString("2.1.3")) &&
+this.olderThan(SparkVersion.fromVersionString("2.2.0")));
Review comment:
I think this works, but if and only if they are evaluated in the given
order, which is a bit error prone. might be worthwhile to add a
`newerThanEqualsPatchVersion` for `2.2.[2+]` type of check
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] felixcheung commented on a change in pull request #3307: ZEPPELIN-4005. SparkRInterpreter is broken for spark 2.1.3 and 2.2.2
felixcheung commented on a change in pull request #3307: ZEPPELIN-4005.
SparkRInterpreter is broken for spark 2.1.3 and 2.2.2
URL: https://github.com/apache/zeppelin/pull/3307#discussion_r257450631
##
File path:
zeppelin-interpreter-integration/src/test/java/org/apache/zeppelin/integration/ZeppelinSparkClusterTest21.java
##
@@ -34,7 +34,7 @@ public ZeppelinSparkClusterTest21(String sparkVersion)
throws Exception {
@Parameterized.Parameters
public static List data() {
return Arrays.asList(new Object[][]{
-{"2.1.2"}
+{"2.1.3"}
Review comment:
can this be passed in thru travis?
and should these be updated
https://github.com/apache/zeppelin/blob/master/.travis.yml#L136 "2.1.0"
https://github.com/apache/zeppelin/blob/master/.travis.yml#L142 "2.2.0"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zjffdu opened a new pull request #3307: ZEPPELIN-4005. SparkRInterpreter is broken for spark 2.1.3 and 2.2.2
zjffdu opened a new pull request #3307: ZEPPELIN-4005. SparkRInterpreter is broken for spark 2.1.3 and 2.2.2 URL: https://github.com/apache/zeppelin/pull/3307 ### What is this PR for? The security patch about the security communication between JVM and R process is backported to spark 2.13 and 2.2.2. And Zeppelin didn't catch that. This PR fix it and also update the test case. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-4005 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4010) CLONE - Remove threshold in log4j.properties
Jeff Zhang created ZEPPELIN-4010: Summary: CLONE - Remove threshold in log4j.properties Key: ZEPPELIN-4010 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4010 Project: Zeppelin Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.9.0 Otherwise user have to change threshold as well if he want to get more debug level logging -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-4009) Large Numbers Truncated
Jason Ferrell created ZEPPELIN-4009:
---
Summary: Large Numbers Truncated
Key: ZEPPELIN-4009
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4009
Project: Zeppelin
Issue Type: Bug
Components: build
Affects Versions: 0.8.0
Environment: %pyspark
from pyspark.sql.functions import *
from pyspark.sql.types import *
sfTestValue = StructField("testValue",StringType(), True)
schemaTest = StructType([sfTestValue])
listTestValues = []
listTestValues.append(("4065453307562594031",))
listTestValues.append(("765995720523059",))
listTestValues.append(("1614560078712787995",))
dfTest = spark.createDataFrame(listTestValues, schemaTest)
dfTestExpanded = dfTest.selectExpr(\"testValue as idAsString",\"cast(testValue
as bigint) as idAsBigint",\"cast(testValue as long) as idAsLong")
dfTestExpanded.show() ##This will show three columns of data correctly.
dfTestExpanded.createOrReplaceGlobalTempView('testTable')
sqlContext.sql('select * from global_temp.testTable').show(3) ##shows truncated
values
Reporter: Jason Ferrell
(Copied from Apache Spark issue 26693 as it appears to be a Zeppelin issue
rather than Spark)
We have a process that takes a file dumped from an external API and formats it
for use in other processes. These API dumps are brought into Spark with all
fields read in as strings. One of the fields is a 19 digit visitor ID. Since
implementing Spark 2.4 a few weeks ago, we have noticed that dataframes read
the 19 digits correctly but any function in SQL appears to truncate the last
two digits and replace them with "00".
Our process is set up to convert these numbers to bigint, which worked before
Spark 2.4. We looked into data types, and the possibility of changing to a
"long" type with no luck. At that point we tried bringing in the string value
as is, with the same result. I've added code that should replicate the issue
with a few 19 digit test cases and demonstrating the type conversions I tried.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #3306: [ZEPPELIN-4001] zeppelin-interpreter-integration is ignored due to wrong folder structure
asfgit closed pull request #3306: [ZEPPELIN-4001] zeppelin-interpreter-integration is ignored due to wrong folder structure URL: https://github.com/apache/zeppelin/pull/3306 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
