[jira] [Commented] (SPARK-22607) Set large stack size consistently for tests to avoid StackOverflowError
[
https://issues.apache.org/jira/browse/SPARK-22607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265810#comment-16265810
]
Apache Spark commented on SPARK-22607:
--
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/19820
> Set large stack size consistently for tests to avoid StackOverflowError
> ---
>
> Key: SPARK-22607
> URL: https://issues.apache.org/jira/browse/SPARK-22607
> Project: Spark
> Issue Type: Bug
> Components: Build, Tests
>Affects Versions: 2.2.0
>Reporter: Sean Owen
>Assignee: Sean Owen
>Priority: Minor
>
> I was seeing this error while testing the 2.2.1 RC:
> {code}
> java.lang.StackOverflowError:
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:370)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> {code}
> This doesn't seem to happen on Jenkins, for whatever reason. It seems like we
> set JVM flags for tests inconsistently, and in particular, only set a 4MB
> stack size for surefire, not scalatest-maven-plugin. Adding {{-Xss4m}} made
> the test pass for me.
> We can also make sure that all of these pass {{-ea}} consistently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-22607) Set large stack size consistently for tests to avoid StackOverflowError
[
https://issues.apache.org/jira/browse/SPARK-22607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-22607:
Assignee: Apache Spark (was: Sean Owen)
> Set large stack size consistently for tests to avoid StackOverflowError
> ---
>
> Key: SPARK-22607
> URL: https://issues.apache.org/jira/browse/SPARK-22607
> Project: Spark
> Issue Type: Bug
> Components: Build, Tests
>Affects Versions: 2.2.0
>Reporter: Sean Owen
>Assignee: Apache Spark
>Priority: Minor
>
> I was seeing this error while testing the 2.2.1 RC:
> {code}
> java.lang.StackOverflowError:
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:370)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> {code}
> This doesn't seem to happen on Jenkins, for whatever reason. It seems like we
> set JVM flags for tests inconsistently, and in particular, only set a 4MB
> stack size for surefire, not scalatest-maven-plugin. Adding {{-Xss4m}} made
> the test pass for me.
> We can also make sure that all of these pass {{-ea}} consistently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-22607) Set large stack size consistently for tests to avoid StackOverflowError
[
https://issues.apache.org/jira/browse/SPARK-22607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-22607:
Assignee: Sean Owen (was: Apache Spark)
> Set large stack size consistently for tests to avoid StackOverflowError
> ---
>
> Key: SPARK-22607
> URL: https://issues.apache.org/jira/browse/SPARK-22607
> Project: Spark
> Issue Type: Bug
> Components: Build, Tests
>Affects Versions: 2.2.0
>Reporter: Sean Owen
>Assignee: Sean Owen
>Priority: Minor
>
> I was seeing this error while testing the 2.2.1 RC:
> {code}
> java.lang.StackOverflowError:
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:370)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> {code}
> This doesn't seem to happen on Jenkins, for whatever reason. It seems like we
> set JVM flags for tests inconsistently, and in particular, only set a 4MB
> stack size for surefire, not scalatest-maven-plugin. Adding {{-Xss4m}} made
> the test pass for me.
> We can also make sure that all of these pass {{-ea}} consistently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22607) Set large stack size consistently for tests to avoid StackOverflowError
Sean Owen created SPARK-22607:
-
Summary: Set large stack size consistently for tests to avoid
StackOverflowError
Key: SPARK-22607
URL: https://issues.apache.org/jira/browse/SPARK-22607
Project: Spark
Issue Type: Bug
Components: Build, Tests
Affects Versions: 2.2.0
Reporter: Sean Owen
Assignee: Sean Owen
Priority: Minor
I was seeing this error while testing the 2.2.1 RC:
{code}
java.lang.StackOverflowError:
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:370)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
{code}
This doesn't seem to happen on Jenkins, for whatever reason. It seems like we
set JVM flags for tests inconsistently, and in particular, only set a 4MB stack
size for surefire, not scalatest-maven-plugin. Adding {{-Xss4m}} made the test
pass for me.
We can also make sure that all of these pass {{-ea}} consistently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22601) Data load is getting displayed successful on providing non existing hdfs file path
[ https://issues.apache.org/jira/browse/SPARK-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265748#comment-16265748 ] Sujith commented on SPARK-22601: sure Sean. > Data load is getting displayed successful on providing non existing hdfs file > path > -- > > Key: SPARK-22601 > URL: https://issues.apache.org/jira/browse/SPARK-22601 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Sujith >Priority: Minor > > Data load is getting displayed successful on providing non existing hdfs file > path where as in local path proper error message is getting displayed > create table tb2 (a string, b int); > load data inpath 'hdfs://hacluster/data1.csv' into table tb2 > Note: data1.csv does not exist in HDFS > when local non existing file path is given below error message will be > displayed > "LOAD DATA input path does not exist". attached snapshots of behaviour in > spark 2.1 and spark 2.2 version -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22402) Allow fetcher URIs to be downloaded to specific locations relative to Mesos Sandbox
[
https://issues.apache.org/jira/browse/SPARK-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265732#comment-16265732
]
Arthur Rand commented on SPARK-22402:
-
Hello [~felixcheung], Yes I'll submit a patch very soon - should be relatively
simple. Thanks.
> Allow fetcher URIs to be downloaded to specific locations relative to Mesos
> Sandbox
> ---
>
> Key: SPARK-22402
> URL: https://issues.apache.org/jira/browse/SPARK-22402
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Arthur Rand
>Priority: Minor
>
> Currently {{spark.mesos.uris}} will only place files in the sandbox, but some
> configuration files and applications may need to be in specific locations.
> The Mesos proto allows for this with the optional {{output_file}} field
> (https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L671).
> We can expose this through the command line with {{--conf
> spark.mesos.uris=:}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos
[ https://issues.apache.org/jira/browse/SPARK-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22583: - Assignee: Kalvin Chau Priority: Major (was: Blocker) > First delegation token renewal time is not 75% of renewal time in Mesos > --- > > Key: SPARK-22583 > URL: https://issues.apache.org/jira/browse/SPARK-22583 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Kalvin Chau >Assignee: Kalvin Chau > Fix For: 2.3.0 > > > The first renewal time of the delegation tokens is the exact renewal time. > This could lead to a situation where the tokens don't make it to executors in > time, and the executors will be working with expired tokens (and Exception > out). > The subsequent renewal times are correctly set to 75% of total renewal time. > The initial renewal time just needs to be set to 75% of the renewal time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos
[ https://issues.apache.org/jira/browse/SPARK-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22583. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19798 [https://github.com/apache/spark/pull/19798] > First delegation token renewal time is not 75% of renewal time in Mesos > --- > > Key: SPARK-22583 > URL: https://issues.apache.org/jira/browse/SPARK-22583 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Kalvin Chau >Priority: Blocker > Fix For: 2.3.0 > > > The first renewal time of the delegation tokens is the exact renewal time. > This could lead to a situation where the tokens don't make it to executors in > time, and the executors will be working with expired tokens (and Exception > out). > The subsequent renewal times are correctly set to 75% of total renewal time. > The initial renewal time just needs to be set to 75% of the renewal time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-22582) Spark SQL round throws error with negative precision
[ https://issues.apache.org/jira/browse/SPARK-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22582. --- Resolution: Not A Problem > Spark SQL round throws error with negative precision > > > Key: SPARK-22582 > URL: https://issues.apache.org/jira/browse/SPARK-22582 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Yuxin Cao > > select round(100.1 , 1) as c3, > round(100.1 , -1) as c5 from orders; > Error: java.lang.IllegalArgumentException: Error: name expected at the > position 10 of 'decimal(4,-1)' but '-' is found. (state=,code=0) > The same query works fine in Spark 1.6. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause
[
https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265666#comment-16265666
]
Mark Petruska commented on SPARK-22393:
---
I believe the scala repl fix would also fix this. Forcing the interpreter to a
non-class based behaviour should also fix this issue (`isClassBased`) in my
opinion. I think the problem is not caused by code and overrides added to
`SparkILoop`. (Sorry about being this vague, but I have some evidence
strengthening the above statements, but nothing that would prove it completely.)
[~rdub] Will do additional tests/experimentation with repl 2.11.8 to confirm;
maybe find a way to force the non-class based behaviour...
> spark-shell can't find imported types in class constructors, extends clause
> ---
>
> Key: SPARK-22393
> URL: https://issues.apache.org/jira/browse/SPARK-22393
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
>Affects Versions: 2.0.2, 2.1.2, 2.2.0
>Reporter: Ryan Williams
>Priority: Minor
>
> {code}
> $ spark-shell
> …
> scala> import org.apache.spark.Partition
> import org.apache.spark.Partition
> scala> class P(p: Partition)
> :11: error: not found: type Partition
>class P(p: Partition)
> ^
> scala> class P(val index: Int) extends Partition
> :11: error: not found: type Partition
>class P(val index: Int) extends Partition
>^
> {code}
> Any class that I {{import}} gives "not found: type ___" when used as a
> parameter to a class, or in an extends clause; this applies to classes I
> import from JARs I provide via {{--jars}} as well as core Spark classes as
> above.
> This worked in 1.6.3 but has been broken since 2.0.0.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-
[ https://issues.apache.org/jira/browse/SPARK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22606: Assignee: (was: Apache Spark) > There may be two or more tasks in one executor will use the same kafka > consumer at the same time, then it will throw an exception: "KafkaConsumer is > not safe for multi-threaded access" > - > > Key: SPARK-22606 > URL: https://issues.apache.org/jira/browse/SPARK-22606 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.2.0 >Reporter: eaton >Priority: Minor > > If the value of param 'spark.streaming.concurrentJobs' is more than one, and > the value of param 'spark.executor.cores' is more than one, there may be two > or more tasks in one executor will use the same kafka consumer at the same > time, then it will throw an exception: "KafkaConsumer is not safe for > multi-threaded access" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi
[ https://issues.apache.org/jira/browse/SPARK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265638#comment-16265638 ] Apache Spark commented on SPARK-22606: -- User 'eatoncys' has created a pull request for this issue: https://github.com/apache/spark/pull/19819 > There may be two or more tasks in one executor will use the same kafka > consumer at the same time, then it will throw an exception: "KafkaConsumer is > not safe for multi-threaded access" > - > > Key: SPARK-22606 > URL: https://issues.apache.org/jira/browse/SPARK-22606 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.2.0 >Reporter: eaton >Priority: Minor > > If the value of param 'spark.streaming.concurrentJobs' is more than one, and > the value of param 'spark.executor.cores' is more than one, there may be two > or more tasks in one executor will use the same kafka consumer at the same > time, then it will throw an exception: "KafkaConsumer is not safe for > multi-threaded access" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-
[ https://issues.apache.org/jira/browse/SPARK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22606: Assignee: Apache Spark > There may be two or more tasks in one executor will use the same kafka > consumer at the same time, then it will throw an exception: "KafkaConsumer is > not safe for multi-threaded access" > - > > Key: SPARK-22606 > URL: https://issues.apache.org/jira/browse/SPARK-22606 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.2.0 >Reporter: eaton >Assignee: Apache Spark >Priority: Minor > > If the value of param 'spark.streaming.concurrentJobs' is more than one, and > the value of param 'spark.executor.cores' is more than one, there may be two > or more tasks in one executor will use the same kafka consumer at the same > time, then it will throw an exception: "KafkaConsumer is not safe for > multi-threaded access" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-t
eaton created SPARK-22606: - Summary: There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access" Key: SPARK-22606 URL: https://issues.apache.org/jira/browse/SPARK-22606 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.2.0 Reporter: eaton Priority: Minor If the value of param 'spark.streaming.concurrentJobs' is more than one, and the value of param 'spark.executor.cores' is more than one, there may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
