[jira] [Commented] (SPARK-15544) Bouncing Zookeeper node causes Active spark master to exit

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796825#comment-16796825
 ] 

Jungtaek Lim commented on SPARK-15544:
--

[~moein7tl]

Do you make some progress? If you are no longer working on this I would like to 
step on it.

> Bouncing Zookeeper node causes Active spark master to exit
> --
>
> Key: SPARK-15544
> URL: https://issues.apache.org/jira/browse/SPARK-15544
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 1.6.1
> Environment: Ubuntu 14.04.  Zookeeper 3.4.6 with 3-node quorum
>Reporter: Steven Lowenthal
>Priority: Major
>
> Shutting Down a single zookeeper node caused spark master to exit.  The 
> master should have connected to a second zookeeper node. 
> {code:title=log output}
> 16/05/25 18:21:28 INFO master.Master: Launching executor 
> app-20160525182128-0006/1 on worker worker-20160524013212-10.16.28.76-59138
> 16/05/25 18:21:28 INFO master.Master: Launching executor 
> app-20160525182128-0006/2 on worker worker-20160524013204-10.16.21.217-47129
> 16/05/26 00:16:01 INFO zookeeper.ClientCnxn: Unable to read additional data 
> from server sessionid 0x154dfc0426b0054, likely server has closed socket, 
> closing socket connection and attempting reconnect
> 16/05/26 00:16:01 INFO zookeeper.ClientCnxn: Unable to read additional data 
> from server sessionid 0x254c701f28d0053, likely server has closed socket, 
> closing socket connection and attempting reconnect
> 16/05/26 00:16:01 INFO state.ConnectionStateManager: State change: SUSPENDED
> 16/05/26 00:16:01 INFO state.ConnectionStateManager: State change: SUSPENDED
> 16/05/26 00:16:01 INFO master.ZooKeeperLeaderElectionAgent: We have lost 
> leadership
> 16/05/26 00:16:01 ERROR master.Master: Leadership has been revoked -- master 
> shutting down. }}
> {code}
> spark-env.sh: 
> {code:title=spark-env.sh}
> export SPARK_LOCAL_DIRS=/ephemeral/spark/local
> export SPARK_WORKER_DIR=/ephemeral/spark/work
> export SPARK_LOG_DIR=/var/log/spark
> export HADOOP_CONF_DIR=/home/ubuntu/hadoop-2.6.3/etc/hadoop
> export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER 
> -Dspark.deploy.zookeeper.url=gn5456-zookeeper-01:2181,gn5456-zookeeper-02:2181,gn5456-zookeeper-03:2181"
> export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true"
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-26739) Standardized Join Types for DataFrames

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-26739:


Assignee: (was: Apache Spark)

> Standardized Join Types for DataFrames
> --
>
> Key: SPARK-26739
> URL: https://issues.apache.org/jira/browse/SPARK-26739
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Skyler Lehan
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> h3. *Q1.* What are you trying to do? Articulate your objectives using 
> absolutely no jargon.
> Currently, in the join functions on 
> [DataFrames|http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset],
>  the join types are defined via a string parameter called joinType. In order 
> for a developer to know which joins are possible, they must look up the API 
> call for join. While this works fine, it can cause the developer to make a 
> typo resulting in improper joins and/or unexpected errors that aren't evident 
> at compile time. The objective of this improvement would be to allow 
> developers to use a common definition for join types (by enum or constants) 
> called JoinTypes. This would contain the possible joins and remove the 
> possibility of a typo. It would also allow Spark to alter the names of the 
> joins in the future without impacting end-users.
> h3. *Q2.* What problem is this proposal NOT designed to solve?
> The problem this solves is extremely narrow, it would not solve anything 
> other than providing a common definition for join types.
> h3. *Q3.* How is it done today, and what are the limits of current practice?
> Currently, developers must join two DataFrames like so:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> "left_outer")
> {code}
> Where they manually type the join type. As stated above, this:
>  * Requires developers to manually type in the join
>  * Can cause possibility of typos
>  * Restricts renaming of join types as its a literal string
>  * Does not restrict and/or compile check the join type being used, leading 
> to runtime errors
> h3. *Q4.* What is new in your approach and why do you think it will be 
> successful?
> The new approach would use constants or *more preferably an enum*, something 
> like this:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> JoinType.LEFT_OUTER)
> {code}
> This would provide:
>  * In code reference/definitions of the possible join types
>  ** This subsequently allows the addition of scaladoc of what each join type 
> does and how it operates
>  * Removes possibilities of a typo on the join type
>  * Provides compile time checking of the join type (only if an enum is used)
> To clarify, if JoinType is a constant, it would just fill in the joinType 
> string parameter for users. If an enum is used, it would restrict the domain 
> of possible join types to whatever is defined in the future JoinType enum. 
> The enum is preferred, however it would take longer to implement.
> h3. *Q5.* Who cares? If you are successful, what difference will it make?
> Developers using Apache Spark will care. This will make the join function 
> easier to wield and lead to less runtime errors. It will save time by 
> bringing join type validation at compile time. It will also provide in code 
> reference to the join types, which saves the developer time of having to look 
> up and navigate the multiple join functions to find the possible join types. 
> In addition to that, the resulting constants/enum would have documentation on 
> how that join type works.
> h3. *Q6.* What are the risks?
> Users of Apache Spark who currently use strings to define their join types 
> could be impacted if an enum is chosen as the common definition. This risk 
> can be mitigated by using string constants. The string constants would be the 
> exact same string as the string literals used today. For example:
> {code:java}
> JoinType.INNER = "inner"
> {code}
> If an enum is still the preferred way of defining the join types, new join 
> functions could be added that take in these enums and the join calls that 
> contain string parameters for joinType could be deprecated. This would give 
> developers a chance to change over to the new join types.
> h3. *Q7.* How long will it take?
> A few days for a seasoned Spark developer.
> h3. *Q8.* What are the mid-term and final "exams" to check for success?
> Mid-term exam would be the addition of a common definition of the join types 
> and additional join functions that take in the join type enum/constant. The 
> final exam would be working tests written to check the functionality of these 
> new join functions and the join functions that take a string for joinType 

[jira] [Assigned] (SPARK-26739) Standardized Join Types for DataFrames

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-26739:


Assignee: Apache Spark

> Standardized Join Types for DataFrames
> --
>
> Key: SPARK-26739
> URL: https://issues.apache.org/jira/browse/SPARK-26739
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Skyler Lehan
>Assignee: Apache Spark
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> h3. *Q1.* What are you trying to do? Articulate your objectives using 
> absolutely no jargon.
> Currently, in the join functions on 
> [DataFrames|http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset],
>  the join types are defined via a string parameter called joinType. In order 
> for a developer to know which joins are possible, they must look up the API 
> call for join. While this works fine, it can cause the developer to make a 
> typo resulting in improper joins and/or unexpected errors that aren't evident 
> at compile time. The objective of this improvement would be to allow 
> developers to use a common definition for join types (by enum or constants) 
> called JoinTypes. This would contain the possible joins and remove the 
> possibility of a typo. It would also allow Spark to alter the names of the 
> joins in the future without impacting end-users.
> h3. *Q2.* What problem is this proposal NOT designed to solve?
> The problem this solves is extremely narrow, it would not solve anything 
> other than providing a common definition for join types.
> h3. *Q3.* How is it done today, and what are the limits of current practice?
> Currently, developers must join two DataFrames like so:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> "left_outer")
> {code}
> Where they manually type the join type. As stated above, this:
>  * Requires developers to manually type in the join
>  * Can cause possibility of typos
>  * Restricts renaming of join types as its a literal string
>  * Does not restrict and/or compile check the join type being used, leading 
> to runtime errors
> h3. *Q4.* What is new in your approach and why do you think it will be 
> successful?
> The new approach would use constants or *more preferably an enum*, something 
> like this:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> JoinType.LEFT_OUTER)
> {code}
> This would provide:
>  * In code reference/definitions of the possible join types
>  ** This subsequently allows the addition of scaladoc of what each join type 
> does and how it operates
>  * Removes possibilities of a typo on the join type
>  * Provides compile time checking of the join type (only if an enum is used)
> To clarify, if JoinType is a constant, it would just fill in the joinType 
> string parameter for users. If an enum is used, it would restrict the domain 
> of possible join types to whatever is defined in the future JoinType enum. 
> The enum is preferred, however it would take longer to implement.
> h3. *Q5.* Who cares? If you are successful, what difference will it make?
> Developers using Apache Spark will care. This will make the join function 
> easier to wield and lead to less runtime errors. It will save time by 
> bringing join type validation at compile time. It will also provide in code 
> reference to the join types, which saves the developer time of having to look 
> up and navigate the multiple join functions to find the possible join types. 
> In addition to that, the resulting constants/enum would have documentation on 
> how that join type works.
> h3. *Q6.* What are the risks?
> Users of Apache Spark who currently use strings to define their join types 
> could be impacted if an enum is chosen as the common definition. This risk 
> can be mitigated by using string constants. The string constants would be the 
> exact same string as the string literals used today. For example:
> {code:java}
> JoinType.INNER = "inner"
> {code}
> If an enum is still the preferred way of defining the join types, new join 
> functions could be added that take in these enums and the join calls that 
> contain string parameters for joinType could be deprecated. This would give 
> developers a chance to change over to the new join types.
> h3. *Q7.* How long will it take?
> A few days for a seasoned Spark developer.
> h3. *Q8.* What are the mid-term and final "exams" to check for success?
> Mid-term exam would be the addition of a common definition of the join types 
> and additional join functions that take in the join type enum/constant. The 
> final exam would be working tests written to check the functionality of these 
> new join functions and the join functions that 

[jira] [Comment Edited] (SPARK-27086) DataSourceV2 MicroBatchExecution commits last batch only if new batch is constructed

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796812#comment-16796812
 ] 

Jungtaek Lim edited comment on SPARK-27086 at 3/20/19 5:38 AM:
---

[~sebastianherold]

This sounds like a kind of request or ask: please use user/dev mailing list for 
that instead.

Btw, MicroBatchReader.commit is not designed to replace checkpoint. It just 
helps for data source to clean up processed offsets. Please refer below:

[https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/MicroBatchReader.java#L70-L74]

The batch can be replayed in case of any chance of failures unless the next 
batch determines the new range of offsets and writes to offset log in 
checkpoint. That's why MicroBatchReader.commit is called later than offset log 
being written. So this works as designed.


was (Author: kabhwan):
[~sebastianherold]

This sounds like a kind of request or ask: please use user/dev mailing list for 
that instead.

Btw, MicroBatchReader.commit is not designed to replace checkpoint. It just 
helps for data source to clean up processed offsets. Please refer below:

[https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/MicroBatchReader.java#L70-L74]

The batch can be replayed in case of any chance of failures unless the next 
batch determines the new range of offsets and writes to offset log in 
checkpoint. That's why MicroBatchReader.commit is called later than offset log 
being written.

> DataSourceV2 MicroBatchExecution commits last batch only if new batch is 
> constructed
> 
>
> Key: SPARK-27086
> URL: https://issues.apache.org/jira/browse/SPARK-27086
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Sebastian Herold
>Priority: Major
>  Labels: MicroBatchExecution, MicroBatchReader, Spark, Streaming, 
> Structured, commit
>
> I wanted to use the new {{DataSourceV2}} API to build a AWS SQS streaming 
> data source which offers the new {{commit}} method of the 
> {{MicroBatchReader}} to finally commit the message at SQS after it has been 
> processed. If the processing of messages would fail and they got not 
> committed, after a timeout the message would automatically reappear in SQS 
> which is the intended behaviour without using special state storing or 
> checkpointing.
> Sadly, I noticed that an offset in the {{MicroBatchReader}} got only 
> committed if a new batch is constructed ([see line 400 in 
> {{MicroBatchExecution}}|https://github.com/apache/spark/blob/f7ad4ff040d39c7a55a9e01a990534e55c8178a5/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L400])
>  which is quite strange. Especially, in my SQS example it could happen that 
> after a first batch of messages this there is a long break before new 
> messages are send to SQS. This would lead to a timeout and reappearance of 
> the SQS messages from the previous batch, because they got processed, but not 
> committed. Therefore, I would strongly recommend to commit an offset, once 
> the batch has been processed! The committing should be independent from the 
> next batch!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27086) DataSourceV2 MicroBatchExecution commits last batch only if new batch is constructed

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796812#comment-16796812
 ] 

Jungtaek Lim commented on SPARK-27086:
--

[~sebastianherold]

This sounds like a kind of request or ask: please use user/dev mailing list for 
that instead.

Btw, MicroBatchReader.commit is not designed to replace checkpoint. It just 
helps for data source to clean up processed offsets. Please refer below:

[https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/MicroBatchReader.java#L70-L74]

The batch can be replayed in case of any chance of failures unless the next 
batch determines the new range of offsets and writes to offset log in 
checkpoint. That's why MicroBatchReader.commit is called later than offset log 
being written.

> DataSourceV2 MicroBatchExecution commits last batch only if new batch is 
> constructed
> 
>
> Key: SPARK-27086
> URL: https://issues.apache.org/jira/browse/SPARK-27086
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Sebastian Herold
>Priority: Major
>  Labels: MicroBatchExecution, MicroBatchReader, Spark, Streaming, 
> Structured, commit
>
> I wanted to use the new {{DataSourceV2}} API to build a AWS SQS streaming 
> data source which offers the new {{commit}} method of the 
> {{MicroBatchReader}} to finally commit the message at SQS after it has been 
> processed. If the processing of messages would fail and they got not 
> committed, after a timeout the message would automatically reappear in SQS 
> which is the intended behaviour without using special state storing or 
> checkpointing.
> Sadly, I noticed that an offset in the {{MicroBatchReader}} got only 
> committed if a new batch is constructed ([see line 400 in 
> {{MicroBatchExecution}}|https://github.com/apache/spark/blob/f7ad4ff040d39c7a55a9e01a990534e55c8178a5/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L400])
>  which is quite strange. Especially, in my SQS example it could happen that 
> after a first batch of messages this there is a long break before new 
> messages are send to SQS. This would lead to a timeout and reappearance of 
> the SQS messages from the previous batch, because they got processed, but not 
> committed. Therefore, I would strongly recommend to commit an offset, once 
> the batch has been processed! The committing should be independent from the 
> next batch!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27209) Split parsing of SELECT and INSERT into two top-level rules in the grammar file.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27209:


Assignee: Apache Spark

> Split parsing of SELECT and INSERT into two top-level rules in the grammar 
> file.
> 
>
> Key: SPARK-27209
> URL: https://issues.apache.org/jira/browse/SPARK-27209
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Dilip Biswal
>Assignee: Apache Spark
>Priority: Major
>
> Currently in the grammar file the rule `query` is responsible to parse both 
> select and insert statements. As a result, we need to have more semantic 
> checks in the code to guard against in valid insert constructs in a query. 
> Couple of examples are in the `visitCreateView` and `visitAlterView` 
> functions. One other issue is that, we don't catch the `invalid insert 
> constructs` in all the places. Here are couple of examples :
> {code:sql}
> select * from (insert into bar values (2));
> {code}
> {code:scala}
> Error in query: unresolved operator 'Project [*];
> 'Project [*]
> +- SubqueryAlias `__auto_generated_subquery_name`
>+- InsertIntoHiveTable `default`.`bar`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
>   +- Project [cast(col1#18 as int) AS c1#20]
>  +- LocalRelation [col1#18]
> {code}
> {code:sql}
> select * from foo where c1 in (insert into bar values (2))
> {code}
> {code:scala}
> Error in query: cannot resolve '(default.foo.`c1` IN (listquery()))' due to 
> data type mismatch: 
> The number of columns in the left hand side of an IN subquery does not match 
> the
> number of columns in the output of subquery.
> #columns in left hand side: 1.
> #columns in right hand side: 0.
> Left side columns:
> [default.foo.`c1`].
> Right side columns:
> [].;;
> 'Project [*]
> +- 'Filter c1#6 IN (list#5 [])
>:  +- InsertIntoHiveTable `default`.`bar`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
>: +- Project [cast(col1#7 as int) AS c1#9]
>:+- LocalRelation [col1#7]
>+- SubqueryAlias `default`.`foo`
>   +- HiveTableRelation `default`.`foo`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#6]
> {code}
> We should have two top-level parser rules to parse `SELECT` and `INSERT` 
> respectively.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27209) Split parsing of SELECT and INSERT into two top-level rules in the grammar file.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27209:


Assignee: (was: Apache Spark)

> Split parsing of SELECT and INSERT into two top-level rules in the grammar 
> file.
> 
>
> Key: SPARK-27209
> URL: https://issues.apache.org/jira/browse/SPARK-27209
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Dilip Biswal
>Priority: Major
>
> Currently in the grammar file the rule `query` is responsible to parse both 
> select and insert statements. As a result, we need to have more semantic 
> checks in the code to guard against in valid insert constructs in a query. 
> Couple of examples are in the `visitCreateView` and `visitAlterView` 
> functions. One other issue is that, we don't catch the `invalid insert 
> constructs` in all the places. Here are couple of examples :
> {code:sql}
> select * from (insert into bar values (2));
> {code}
> {code:scala}
> Error in query: unresolved operator 'Project [*];
> 'Project [*]
> +- SubqueryAlias `__auto_generated_subquery_name`
>+- InsertIntoHiveTable `default`.`bar`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
>   +- Project [cast(col1#18 as int) AS c1#20]
>  +- LocalRelation [col1#18]
> {code}
> {code:sql}
> select * from foo where c1 in (insert into bar values (2))
> {code}
> {code:scala}
> Error in query: cannot resolve '(default.foo.`c1` IN (listquery()))' due to 
> data type mismatch: 
> The number of columns in the left hand side of an IN subquery does not match 
> the
> number of columns in the output of subquery.
> #columns in left hand side: 1.
> #columns in right hand side: 0.
> Left side columns:
> [default.foo.`c1`].
> Right side columns:
> [].;;
> 'Project [*]
> +- 'Filter c1#6 IN (list#5 [])
>:  +- InsertIntoHiveTable `default`.`bar`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
>: +- Project [cast(col1#7 as int) AS c1#9]
>:+- LocalRelation [col1#7]
>+- SubqueryAlias `default`.`foo`
>   +- HiveTableRelation `default`.`foo`, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#6]
> {code}
> We should have two top-level parser rules to parse `SELECT` and `INSERT` 
> respectively.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27209) Split parsing of SELECT and INSERT into two top-level rules in the grammar file.

[Dilip Biswal (JIRA)]

Dilip Biswal created SPARK-27209:


 Summary: Split parsing of SELECT and INSERT into two top-level 
rules in the grammar file.
 Key: SPARK-27209
 URL: https://issues.apache.org/jira/browse/SPARK-27209
 Project: Spark
  Issue Type: Improvement
  Components: SQL
Affects Versions: 2.4.0
Reporter: Dilip Biswal


Currently in the grammar file the rule `query` is responsible to parse both 
select and insert statements. As a result, we need to have more semantic checks 
in the code to guard against in valid insert constructs in a query. Couple of 
examples are in the `visitCreateView` and `visitAlterView` functions. One other 
issue is that, we don't catch the `invalid insert constructs` in all the 
places. Here are couple of examples :

{code:sql}
select * from (insert into bar values (2));
{code}
{code:scala}
Error in query: unresolved operator 'Project [*];
'Project [*]
+- SubqueryAlias `__auto_generated_subquery_name`
   +- InsertIntoHiveTable `default`.`bar`, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
  +- Project [cast(col1#18 as int) AS c1#20]
 +- LocalRelation [col1#18]
{code}

{code:sql}
select * from foo where c1 in (insert into bar values (2))
{code}
{code:scala}
Error in query: cannot resolve '(default.foo.`c1` IN (listquery()))' due to 
data type mismatch: 
The number of columns in the left hand side of an IN subquery does not match the
number of columns in the output of subquery.
#columns in left hand side: 1.
#columns in right hand side: 0.

Left side columns:
[default.foo.`c1`].
Right side columns:
[].;;
'Project [*]
+- 'Filter c1#6 IN (list#5 [])
   :  +- InsertIntoHiveTable `default`.`bar`, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1]
   : +- Project [cast(col1#7 as int) AS c1#9]
   :+- LocalRelation [col1#7]
   +- SubqueryAlias `default`.`foo`
  +- HiveTableRelation `default`.`foo`, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#6]
{code}

We should have two top-level parser rules to parse `SELECT` and `INSERT` 
respectively.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-16190) Worker registration failed: Duplicate worker ID

[zuotingbing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796772#comment-16796772
 ] 

zuotingbing commented on SPARK-16190:
-

i faced the same issue. worker log as follows:
{code:java}
// code placeholder
{code}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 2019-03-15 20:22:14,862 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,863 
INFO Worker: Connecting to master vmax18:7077... 2019-03-15 20:22:14,863 INFO 
Worker: Connecting to master vmax17:7077... 2019-03-15 20:22:14,865 INFO 
Worker: Master with url spark://vmax18:7077 requested this worker to reconnect. 
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 2019-03-15 
20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 requested this 
worker to reconnect. 2019-03-15 20:22:14,868 INFO Worker: Not spawning another 
attempt to register with the master, since there is an attempt scheduled 
already. 2019-03-15 20:22:14,871 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,871 
INFO Worker: Not spawning another attempt to register with the master, since 
there is an attempt scheduled already. 2019-03-15 20:22:14,879 ERROR Worker: 
Worker registration failed: Duplicate worker ID 2019-03-15 20:22:14,891 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,891 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,894 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,895 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,895 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,895 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,896 INFO ShutdownHookManager: Shutdown hook called 
2019-03-15 20:22:14,898 INFO ShutdownHookManager: Deleting directory 
/data4/zdh/spark/tmp/spark-c578bf32-6a5e-44a5-843b-c796f44648ee 2019-03-15 
20:22:14,908 INFO ShutdownHookManager: Deleting directory 
/data3/zdh/spark/tmp/spark-7e57e77d-cbb7-47d3-a6dd-737b57788533 2019-03-15 
20:22:14,920 INFO ShutdownHookManager: Deleting directory 
/data2/zdh/spark/tmp/spark-0beebf20-abbd-4d99-a401-3ef0e88e0b05

> Worker registration failed: Duplicate worker ID
> ---
>
> Key: SPARK-16190
> URL: https://issues.apache.org/jira/browse/SPARK-16190
> Project: Spark
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: 1.6.1
>Reporter: Thomas Huang
>Priority: Minor
> Attachments: 
> spark-mqq-org.apache.spark.deploy.worker.Worker-1-slave19.out, 
> spark-mqq-org.apache.spark.deploy.worker.Worker-1-slave2.out, 
> spark-mqq-org.apache.spark.deploy.worker.Worker-1-slave7.out, 
> spark-mqq-org.apache.spark.deploy.worker.Worker-1-slave8.out
>
>
> Several worker crashed simultaneously due to this error: 
> Worker registration failed: Duplicate worker ID
> This is the worker log on one of those crashed workers:
> 16/06/24 16:28:53 INFO ExecutorRunner: Killing process!
> 16/06/24 16:28:53 INFO ExecutorRunner: Runner thread for executor 
> app-20160624003013-0442/26 interrupted
> 16/06/24 16:28:53 INFO ExecutorRunner: Killing process!
> 16/06/24 16:29:03 WARN ExecutorRunner: Failed to terminate process: 
> java.lang.UNIXProcess@31340137. This process will likely be orphaned.
> 16/06/24 16:29:03 WARN ExecutorRunner: Failed to terminate process: 
> java.lang.UNIXProcess@4d3bdb1d. This process will likely be orphaned.
> 16/06/24 16:29:03 INFO Worker: Executor app-20160624003013-0442/8 finished 
> with state KILLED
> 16/06/24 16:29:03 INFO Worker: Executor app-20160624003013-0442/26 finished 
> with state KILLED
> 16/06/24 16:29:03 INFO Worker: Cleaning up local directories for application 
> app-20160624003013-0442
> 16/06/24 16:31:18 INFO ExternalShuffleBlockResolver: Application 
> app-20160624003013-0442 removed, cleanupLocalDirs = true
> 16/06/24 16:31:18 INFO Worker: Asked to launch executor 
> 

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[zuotingbing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796772#comment-16796772
 ] 

zuotingbing edited comment on SPARK-16190 at 3/20/19 3:57 AM:
--

i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 2019-03-15 20:22:14,862 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,863 
INFO Worker: Connecting to master vmax18:7077... 2019-03-15 20:22:14,863 INFO 
Worker: Connecting to master vmax17:7077... 2019-03-15 20:22:14,865 INFO 
Worker: Master with url spark://vmax18:7077 requested this worker to reconnect. 
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 2019-03-15 
20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 requested this 
worker to reconnect. 2019-03-15 20:22:14,868 INFO Worker: Not spawning another 
attempt to register with the master, since there is an attempt scheduled 
already. 2019-03-15 20:22:14,871 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,871 
INFO Worker: Not spawning another attempt to register with the master, since 
there is an attempt scheduled already. 2019-03-15 20:22:14,879 ERROR Worker: 
Worker registration failed: Duplicate worker ID 2019-03-15 20:22:14,891 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,891 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,894 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,895 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,895 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,895 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,896 INFO ShutdownHookManager: Shutdown hook called 
2019-03-15 20:22:14,898 INFO ShutdownHookManager: Deleting directory 
/data4/zdh/spark/tmp/spark-c578bf32-6a5e-44a5-843b-c796f44648ee 2019-03-15 
20:22:14,908 INFO ShutdownHookManager: Deleting directory 
/data3/zdh/spark/tmp/spark-7e57e77d-cbb7-47d3-a6dd-737b57788533 2019-03-15 
20:22:14,920 INFO ShutdownHookManager: Deleting directory 
/data2/zdh/spark/tmp/spark-0beebf20-abbd-4d99-a401-3ef0e88e0b05{code}
 


was (Author: zuo.tingbing9):
i faced the same issue. worker log as follows:
{code:java}
// code placeholder
{code}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 2019-03-15 20:22:14,862 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,863 
INFO Worker: Connecting to master vmax18:7077... 2019-03-15 20:22:14,863 INFO 
Worker: Connecting to master vmax17:7077... 2019-03-15 20:22:14,865 INFO 
Worker: Master with url spark://vmax18:7077 requested this worker to reconnect. 
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 2019-03-15 
20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 requested this 
worker to reconnect. 2019-03-15 20:22:14,868 INFO Worker: Not spawning another 
attempt to register with the master, since there is an attempt scheduled 
already. 2019-03-15 20:22:14,871 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,871 
INFO Worker: Not spawning another attempt to register with the master, since 
there is an attempt scheduled already. 2019-03-15 20:22:14,879 ERROR Worker: 
Worker registration failed: Duplicate worker ID 2019-03-15 20:22:14,891 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,891 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[zuotingbing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796772#comment-16796772
 ] 

zuotingbing edited comment on SPARK-16190 at 3/20/19 4:02 AM:
--

i faced the same issue in standalone HA mode. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 
2019-03-15 20:22:14,862 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax18:7077... 
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax17:7077... 
2019-03-15 20:22:14,865 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,868 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,871 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,871 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,879 ERROR Worker: Worker registration failed: Duplicate 
worker ID
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,896 INFO ShutdownHookManager: Shutdown hook called 
2019-03-15 20:22:14,898 INFO ShutdownHookManager: Deleting directory 
/data4/zdh/spark/tmp/spark-c578bf32-6a5e-44a5-843b-c796f44648ee 
2019-03-15 20:22:14,908 INFO ShutdownHookManager: Deleting directory 
/data3/zdh/spark/tmp/spark-7e57e77d-cbb7-47d3-a6dd-737b57788533 
2019-03-15 20:22:14,920 INFO ShutdownHookManager: Deleting directory 
/data2/zdh/spark/tmp/spark-0beebf20-abbd-4d99-a401-3ef0e88e0b05{code}
 


was (Author: zuo.tingbing9):
i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 
2019-03-15 20:22:14,862 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax18:7077... 
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax17:7077... 
2019-03-15 20:22:14,865 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,868 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,871 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,871 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,879 ERROR Worker: Worker registration failed: Duplicate 
worker ID
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[zuotingbing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796772#comment-16796772
 ] 

zuotingbing edited comment on SPARK-16190 at 3/20/19 3:59 AM:
--

i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 
2019-03-15 20:22:14,862 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax18:7077... 
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax17:7077... 
2019-03-15 20:22:14,865 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,868 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,871 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,871 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,879 ERROR Worker: Worker registration failed: Duplicate 
worker ID 2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,896 INFO ShutdownHookManager: Shutdown hook called 
2019-03-15 20:22:14,898 INFO ShutdownHookManager: Deleting directory 
/data4/zdh/spark/tmp/spark-c578bf32-6a5e-44a5-843b-c796f44648ee 
2019-03-15 20:22:14,908 INFO ShutdownHookManager: Deleting directory 
/data3/zdh/spark/tmp/spark-7e57e77d-cbb7-47d3-a6dd-737b57788533 
2019-03-15 20:22:14,920 INFO ShutdownHookManager: Deleting directory 
/data2/zdh/spark/tmp/spark-0beebf20-abbd-4d99-a401-3ef0e88e0b05{code}
 


was (Author: zuo.tingbing9):
i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 2019-03-15 20:22:14,862 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,863 
INFO Worker: Connecting to master vmax18:7077... 2019-03-15 20:22:14,863 INFO 
Worker: Connecting to master vmax17:7077... 2019-03-15 20:22:14,865 INFO 
Worker: Master with url spark://vmax18:7077 requested this worker to reconnect. 
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 2019-03-15 
20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 requested this 
worker to reconnect. 2019-03-15 20:22:14,868 INFO Worker: Not spawning another 
attempt to register with the master, since there is an attempt scheduled 
already. 2019-03-15 20:22:14,871 INFO Worker: Master with url 
spark://vmax18:7077 requested this worker to reconnect. 2019-03-15 20:22:14,871 
INFO Worker: Not spawning another attempt to register with the master, since 
there is an attempt scheduled already. 2019-03-15 20:22:14,879 ERROR Worker: 
Worker registration failed: Duplicate worker ID 2019-03-15 20:22:14,891 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,891 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 
20:22:14,893 INFO ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO 
ExecutorRunner: Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: 
Killing process! 2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: 

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[zuotingbing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796772#comment-16796772
 ] 

zuotingbing edited comment on SPARK-16190 at 3/20/19 4:00 AM:
--

i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 
2019-03-15 20:22:14,862 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax18:7077... 
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax17:7077... 
2019-03-15 20:22:14,865 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,868 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,871 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,871 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,879 ERROR Worker: Worker registration failed: Duplicate 
worker ID
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,895 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,896 INFO ShutdownHookManager: Shutdown hook called 
2019-03-15 20:22:14,898 INFO ShutdownHookManager: Deleting directory 
/data4/zdh/spark/tmp/spark-c578bf32-6a5e-44a5-843b-c796f44648ee 
2019-03-15 20:22:14,908 INFO ShutdownHookManager: Deleting directory 
/data3/zdh/spark/tmp/spark-7e57e77d-cbb7-47d3-a6dd-737b57788533 
2019-03-15 20:22:14,920 INFO ShutdownHookManager: Deleting directory 
/data2/zdh/spark/tmp/spark-0beebf20-abbd-4d99-a401-3ef0e88e0b05{code}
 


was (Author: zuo.tingbing9):
i faced the same issue. worker log as follows:
{code:java}
2019-03-15 20:22:10,474 INFO Worker: Master has changed, new master is at 
spark://vmax17:7077 
2019-03-15 20:22:14,862 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax18:7077... 
2019-03-15 20:22:14,863 INFO Worker: Connecting to master vmax17:7077... 
2019-03-15 20:22:14,865 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect.
2019-03-15 20:22:14,865 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,868 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,868 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,871 INFO Worker: Master with url spark://vmax18:7077 
requested this worker to reconnect. 
2019-03-15 20:22:14,871 INFO Worker: Not spawning another attempt to register 
with the master, since there is an attempt scheduled already. 
2019-03-15 20:22:14,879 ERROR Worker: Worker registration failed: Duplicate 
worker ID 2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,891 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,893 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO ExecutorRunner: Killing process! 
2019-03-15 20:22:14,894 INFO 

[jira] [Commented] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

[Dongjoon Hyun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796760#comment-16796760
 ] 

Dongjoon Hyun commented on SPARK-27160:
---

Hi, [~sadhen]. Thank you so much for this contribution. Next time, please don't 
set the `Fix Versions` and don't use `Blocker` priority. That is correct for 
this issue, but there is a guide line for contribution in Apache Spark 
community. In general, we had better respect the guideline.
- https://spark.apache.org/contributing.html

> Incorrect Literal Casting of DecimalType in OrcFilters
> --
>
> Key: SPARK-27160
> URL: https://issues.apache.org/jira/browse/SPARK-27160
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Darcy Shen
>Priority: Blocker
>  Labels: correctness
> Fix For: 3.0.0
>
>
> DecimalType Literal should not be casted to Long.
> eg. For `df.filter("x < 3.14")`, assuming df (x in DecimalType) reads from a 
> ORC table and uses the native ORC reader with predicate push down enabled, we 
> will push down the `x < 3.14` predicate to the ORC reader via a 
> SearchArgument.
> OrcFilters will construct the SearchArgument, but not handle the DecimalType 
> correctly.
> The previous impl will construct `x < 3` from `x < 3.14`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

[Dongjoon Hyun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-27160:
--
Fix Version/s: (was: 2.4.1)

> Incorrect Literal Casting of DecimalType in OrcFilters
> --
>
> Key: SPARK-27160
> URL: https://issues.apache.org/jira/browse/SPARK-27160
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Darcy Shen
>Priority: Blocker
>  Labels: correctness
> Fix For: 3.0.0
>
>
> DecimalType Literal should not be casted to Long.
> eg. For `df.filter("x < 3.14")`, assuming df (x in DecimalType) reads from a 
> ORC table and uses the native ORC reader with predicate push down enabled, we 
> will push down the `x < 3.14` predicate to the ORC reader via a 
> SearchArgument.
> OrcFilters will construct the SearchArgument, but not handle the DecimalType 
> correctly.
> The previous impl will construct `x < 3` from `x < 3.14`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26975) Support nested-column pruning over limit/sample/repartition

[Dongjoon Hyun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun resolved SPARK-26975.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

This is resolved via https://github.com/apache/spark/pull/23964

> Support nested-column pruning over limit/sample/repartition
> ---
>
> Key: SPARK-26975
> URL: https://issues.apache.org/jira/browse/SPARK-26975
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.0.0
>
>
> As SPARK-26958 shows the benchmark, nested-column pruning has limitations. 
> This issue aims to remove the limitations on `limit/repartition/sample`. In 
> this issue, repartition means `Repartition`, not `RepartitionByExpression`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[Imran Rashid (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Imran Rashid updated SPARK-27112:
-
Fix Version/s: (was: 2.4.1)
   2.4.2

> Spark Scheduler encounters two independent Deadlocks when trying to kill 
> executors either due to dynamic allocation or blacklisting 
> 
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
>  Issue Type: Bug
>  Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.2, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot 
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png, 
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs 
> were getting stuck. On further analysis, it was found out that there exist 
> two independent deadlocks and either of them occur under different 
> circumstances. The screenshots for these two deadlocks are attached here. 
> We were able to reproduce the deadlocks with the following piece of code:
>  
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount  ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 && 
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for 
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>  
> Additionally, to ensure that the deadlock surfaces up soon enough, I also 
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>  
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>  
> Also make sure that the following configs are set when launching the above 
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[Imran Rashid (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796677#comment-16796677
 ] 

Imran Rashid commented on SPARK-27112:
--

I revised the fix version for 2.4 to 2.4.2, as I just realized rc8 was cut 
before this was committed.  If there is another rc, we should adjust the fix 
version.

> Spark Scheduler encounters two independent Deadlocks when trying to kill 
> executors either due to dynamic allocation or blacklisting 
> 
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
>  Issue Type: Bug
>  Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.2, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot 
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png, 
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs 
> were getting stuck. On further analysis, it was found out that there exist 
> two independent deadlocks and either of them occur under different 
> circumstances. The screenshots for these two deadlocks are attached here. 
> We were able to reproduce the deadlocks with the following piece of code:
>  
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount  ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 && 
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for 
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>  
> Additionally, to ensure that the deadlock surfaces up soon enough, I also 
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>  
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>  
> Also make sure that the following configs are set when launching the above 
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27186) Optimize SortShuffleWriter writing process

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796671#comment-16796671
 ] 

Apache Spark commented on SPARK-27186:
--

User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/24124

> Optimize SortShuffleWriter writing process
> --
>
> Key: SPARK-27186
> URL: https://issues.apache.org/jira/browse/SPARK-27186
> Project: Spark
>  Issue Type: Improvement
>  Components: Shuffle
>Affects Versions: 3.0.0
>Reporter: wangjiaochun
>Priority: Minor
>
> If the SortShuffleWriter.write method records is empty, it should be return 
> directly and no need to keep running，refer to the process of 
> BypassMergeSortShuffleWriter.write. Here are the benefits of create Instance 
> ExternalSorter and tmp file. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26736) if filter condition `And` has non-determined sub function it does not do partition prunning

[roncenzhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

roncenzhao updated SPARK-26736:
---
Summary: if filter condition `And` has non-determined sub function it does 
not do partition prunning  (was: if filter condition has rand() function it 
does not do partition prunning)

> if filter condition `And` has non-determined sub function it does not do 
> partition prunning
> ---
>
> Key: SPARK-26736
> URL: https://issues.apache.org/jira/browse/SPARK-26736
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: roncenzhao
>Priority: Minor
>
> Example:
> A  partitioned table definition:
> _create table test(id int) partitioned by (dt string);_
> The following sql does not do partition prunning:
> _select * from test where dt='20190101' and rand() < 0.5;_
>  
> I think it should do partition prunning in this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-19254) Support Seq, Map, and Struct in functions.lit

[Artem Rukavytsia (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753582#comment-16753582
 ] 

Artem Rukavytsia edited comment on SPARK-19254 at 3/20/19 1:03 AM:
---

Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. This behavior is unexpected.


was (Author: arukavytsia):
Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. This is unexpected behavior.

> Support Seq, Map, and Struct in functions.lit
> -
>
> Key: SPARK-19254
> URL: https://issues.apache.org/jira/browse/SPARK-19254
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Takeshi Yamamuro
>Assignee: Takeshi Yamamuro
>Priority: Minor
> Fix For: 2.2.0
>
>
> In the current implementation, function.lit does not support Seq, Map, and 
> Struct. This ticket is intended to support them. This is the follow-up of 
> https://issues.apache.org/jira/browse/SPARK-17683.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-19254) Support Seq, Map, and Struct in functions.lit

[Artem Rukavytsia (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753582#comment-16753582
 ] 

Artem Rukavytsia edited comment on SPARK-19254 at 3/20/19 1:02 AM:
---

Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. This is unexpected behavior.


was (Author: arukavytsia):
Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. Suggest, it's the unexpected behavior. 

> Support Seq, Map, and Struct in functions.lit
> -
>
> Key: SPARK-19254
> URL: https://issues.apache.org/jira/browse/SPARK-19254
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Takeshi Yamamuro
>Assignee: Takeshi Yamamuro
>Priority: Minor
> Fix For: 2.2.0
>
>
> In the current implementation, function.lit does not support Seq, Map, and 
> Struct. This ticket is intended to support them. This is the follow-up of 
> https://issues.apache.org/jira/browse/SPARK-17683.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-19254) Support Seq, Map, and Struct in functions.lit

[Artem Rukavytsia (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753582#comment-16753582
 ] 

Artem Rukavytsia edited comment on SPARK-19254 at 3/20/19 1:00 AM:
---

Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. Suggest, it's the unexpected behavior. 


was (Author: arukavytsia):
Looks like Pyspark doesn't support `typedLit`, `from pyspark.sql.functions 
import typedLit` is failing. Suggest it's the unexpected behavior. 

> Support Seq, Map, and Struct in functions.lit
> -
>
> Key: SPARK-19254
> URL: https://issues.apache.org/jira/browse/SPARK-19254
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Takeshi Yamamuro
>Assignee: Takeshi Yamamuro
>Priority: Minor
> Fix For: 2.2.0
>
>
> In the current implementation, function.lit does not support Seq, Map, and 
> Struct. This ticket is intended to support them. This is the follow-up of 
> https://issues.apache.org/jira/browse/SPARK-17683.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-27197) Add ReadNestedSchemaTest for file-based data sources

[DB Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DB Tsai resolved SPARK-27197.
-
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 24139
[https://github.com/apache/spark/pull/24139]

> Add ReadNestedSchemaTest for file-based data sources
> 
>
> Key: SPARK-27197
> URL: https://issues.apache.org/jira/browse/SPARK-27197
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL, Tests
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue adds a test coverage into the schema evolution suite for adding 
> nested column and hiding nested columns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21097) Dynamic allocation will preserve cached data

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin reassigned SPARK-21097:
--

Assignee: (was: Marcelo Vanzin)

> Dynamic allocation will preserve cached data
> 
>
> Key: SPARK-21097
> URL: https://issues.apache.org/jira/browse/SPARK-21097
> Project: Spark
>  Issue Type: Improvement
>  Components: Block Manager, Scheduler, Spark Core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Brad
>Priority: Major
> Attachments: Preserving Cached Data with Dynamic Allocation.pdf
>
>
> We want to use dynamic allocation to distribute resources among many notebook 
> users on our spark clusters. One difficulty is that if a user has cached data 
> then we are either prevented from de-allocating any of their executors, or we 
> are forced to drop their cached data, which can lead to a bad user experience.
> We propose adding a feature to preserve cached data by copying it to other 
> executors before de-allocation. This behavior would be enabled by a simple 
> spark config. Now when an executor reaches its configured idle timeout, 
> instead of just killing it on the spot, we will stop sending it new tasks, 
> replicate all of its rdd blocks onto other executors, and then kill it. If 
> there is an issue while we replicate the data, like an error, it takes too 
> long, or there isn't enough space, then we will fall back to the original 
> behavior and drop the data and kill the executor.
> This feature should allow anyone with notebook users to use their cluster 
> resources more efficiently. Also since it will be completely opt-in it will 
> unlikely to cause problems for other use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21097) Dynamic allocation will preserve cached data

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin reassigned SPARK-21097:
--

Assignee: Marcelo Vanzin

> Dynamic allocation will preserve cached data
> 
>
> Key: SPARK-21097
> URL: https://issues.apache.org/jira/browse/SPARK-21097
> Project: Spark
>  Issue Type: Improvement
>  Components: Block Manager, Scheduler, Spark Core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Brad
>Assignee: Marcelo Vanzin
>Priority: Major
> Attachments: Preserving Cached Data with Dynamic Allocation.pdf
>
>
> We want to use dynamic allocation to distribute resources among many notebook 
> users on our spark clusters. One difficulty is that if a user has cached data 
> then we are either prevented from de-allocating any of their executors, or we 
> are forced to drop their cached data, which can lead to a bad user experience.
> We propose adding a feature to preserve cached data by copying it to other 
> executors before de-allocation. This behavior would be enabled by a simple 
> spark config. Now when an executor reaches its configured idle timeout, 
> instead of just killing it on the spot, we will stop sending it new tasks, 
> replicate all of its rdd blocks onto other executors, and then kill it. If 
> there is an issue while we replicate the data, like an error, it takes too 
> long, or there isn't enough space, then we will fall back to the original 
> behavior and drop the data and kill the executor.
> This feature should allow anyone with notebook users to use their cluster 
> resources more efficiently. Also since it will be completely opt-in it will 
> unlikely to cause problems for other use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin updated SPARK-27194:
---
Description: 
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:
{code}
19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.
{code}

The task is re-attempted on a different executor and fails because the 
part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
already exists:
{code}
19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
(TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:109)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client a.b.c.d already exists
{code}

The job fails when the the configured task attempts (spark.task.maxFailures) 
have failed with the same error:
{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 in 
stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 0.0 
(TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 ...
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client i.p.a.d already exists
{code}

SPARK-26682 wasn't the root cause here, since there wasn't any stage reattempt.

This issue seems to happen when 
spark.sql.sources.partitionOverwriteMode=dynamic. 

 

  was:
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:
{code}
19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 

[jira] [Created] (SPARK-27208) RestSubmissionClient only supports http

[Jorge Machado (JIRA)]

Jorge Machado created SPARK-27208:
-

 Summary: RestSubmissionClient only supports http
 Key: SPARK-27208
 URL: https://issues.apache.org/jira/browse/SPARK-27208
 Project: Spark
  Issue Type: Bug
  Components: Mesos
Affects Versions: 2.4.0
Reporter: Jorge Machado


As stand of now the class RestSubmissionClient does not support https, which 
fails for example if we run mesos master with ssl and in cluster mode. 

The spark-submit command fails with: Mesos cluster mode is only supported 
through the REST submission API

 

I create a PR for this which checks if the master endpoint given can speak ssl 
before submitting the command. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27207) There exists a bug with SortBasedAggregator where merge()/update() operations get invoked on the aggregate buffer without calling initialize

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27207:


Assignee: (was: Apache Spark)

> There exists a bug with SortBasedAggregator where merge()/update() operations 
> get invoked on the aggregate buffer without calling initialize
> 
>
> Key: SPARK-27207
> URL: https://issues.apache.org/jira/browse/SPARK-27207
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Parth Gandhi
>Priority: Minor
>
> Normally, the aggregate operations that are invoked for an aggregation buffer 
> for User Defined Aggregate Functions(UDAF) follow the order like 
> initialize(), update(), eval() OR initialize(), merge(), eval(). However, 
> after a certain threshold configurable by 
> spark.sql.objectHashAggregate.sortBased.fallbackThreshold is reached, 
> ObjectHashAggregate falls back to SortBasedAggregator which invokes the merge 
> or update operation without calling initialize() on the aggregate buffer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27207) There exists a bug with SortBasedAggregator where merge()/update() operations get invoked on the aggregate buffer without calling initialize

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27207:


Assignee: Apache Spark

> There exists a bug with SortBasedAggregator where merge()/update() operations 
> get invoked on the aggregate buffer without calling initialize
> 
>
> Key: SPARK-27207
> URL: https://issues.apache.org/jira/browse/SPARK-27207
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Parth Gandhi
>Assignee: Apache Spark
>Priority: Minor
>
> Normally, the aggregate operations that are invoked for an aggregation buffer 
> for User Defined Aggregate Functions(UDAF) follow the order like 
> initialize(), update(), eval() OR initialize(), merge(), eval(). However, 
> after a certain threshold configurable by 
> spark.sql.objectHashAggregate.sortBased.fallbackThreshold is reached, 
> ObjectHashAggregate falls back to SortBasedAggregator which invokes the merge 
> or update operation without calling initialize() on the aggregate buffer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796544#comment-16796544
 ] 

Jorge Machado commented on SPARK-27208:
---

relates to this as the follow up error

> RestSubmissionClient only supports http
> ---
>
> Key: SPARK-27208
> URL: https://issues.apache.org/jira/browse/SPARK-27208
> Project: Spark
>  Issue Type: Bug
>  Components: Mesos
>Affects Versions: 2.4.0
>Reporter: Jorge Machado
>Priority: Minor
>
> As stand of now the class RestSubmissionClient does not support https, which 
> fails for example if we run mesos master with ssl and in cluster mode. 
> The spark-submit command fails with: Mesos cluster mode is only supported 
> through the REST submission API
>  
> I create a PR for this which checks if the master endpoint given can speak 
> ssl before submitting the command. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[Imran Rashid (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Imran Rashid updated SPARK-27112:
-
Fix Version/s: 2.4.1
   2.3.4

> Spark Scheduler encounters two independent Deadlocks when trying to kill 
> executors either due to dynamic allocation or blacklisting 
> 
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
>  Issue Type: Bug
>  Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.1, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot 
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png, 
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs 
> were getting stuck. On further analysis, it was found out that there exist 
> two independent deadlocks and either of them occur under different 
> circumstances. The screenshots for these two deadlocks are attached here. 
> We were able to reproduce the deadlocks with the following piece of code:
>  
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount  ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 && 
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for 
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>  
> Additionally, to ensure that the deadlock surfaces up soon enough, I also 
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>  
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>  
> Also make sure that the following configs are set when launching the above 
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[Imran Rashid (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796539#comment-16796539
 ] 

Imran Rashid commented on SPARK-27112:
--

Fixed in 2.4 & 2.3 by https://github.com/apache/spark/pull/24134

> Spark Scheduler encounters two independent Deadlocks when trying to kill 
> executors either due to dynamic allocation or blacklisting 
> 
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
>  Issue Type: Bug
>  Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.1, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot 
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png, 
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs 
> were getting stuck. On further analysis, it was found out that there exist 
> two independent deadlocks and either of them occur under different 
> circumstances. The screenshots for these two deadlocks are attached here. 
> We were able to reproduce the deadlocks with the following piece of code:
>  
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount  ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 && 
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for 
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>  
> Additionally, to ensure that the deadlock surfaces up soon enough, I also 
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>  
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>  
> Also make sure that the following configs are set when launching the above 
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

[Imran Rashid (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Imran Rashid resolved SPARK-26288.
--
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 23393
[https://github.com/apache/spark/pull/23393]

> add initRegisteredExecutorsDB in ExternalShuffleService
> ---
>
> Key: SPARK-26288
> URL: https://issues.apache.org/jira/browse/SPARK-26288
> Project: Spark
>  Issue Type: New Feature
>  Components: Kubernetes, Shuffle, Spark Core
>Affects Versions: 2.4.0
>Reporter: weixiuli
>Assignee: weixiuli
>Priority: Major
> Fix For: 3.0.0
>
>
> As we all know that spark on Yarn uses DB to record RegisteredExecutors 
> information which can be reloaded and used again when the 
> ExternalShuffleService is restarted .
> The RegisteredExecutors information can't be recorded both in the mode of 
> spark's standalone and spark on k8s , which will cause the 
> RegisteredExecutors information to be lost ,when the ExternalShuffleService 
> is restarted.
> To solve the problem above, a method is proposed and is committed .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

[Imran Rashid (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Imran Rashid reassigned SPARK-26288:


Assignee: weixiuli

> add initRegisteredExecutorsDB in ExternalShuffleService
> ---
>
> Key: SPARK-26288
> URL: https://issues.apache.org/jira/browse/SPARK-26288
> Project: Spark
>  Issue Type: New Feature
>  Components: Kubernetes, Shuffle, Spark Core
>Affects Versions: 2.4.0
>Reporter: weixiuli
>Assignee: weixiuli
>Priority: Major
>
> As we all know that spark on Yarn uses DB to record RegisteredExecutors 
> information which can be reloaded and used again when the 
> ExternalShuffleService is restarted .
> The RegisteredExecutors information can't be recorded both in the mode of 
> spark's standalone and spark on k8s , which will cause the 
> RegisteredExecutors information to be lost ,when the ExternalShuffleService 
> is restarted.
> To solve the problem above, a method is proposed and is committed .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27207) There exists a bug with SortBasedAggregator where merge()/update() operations get invoked on the aggregate buffer without calling initialize

[Parth Gandhi (JIRA)]

Parth Gandhi created SPARK-27207:


 Summary: There exists a bug with SortBasedAggregator where 
merge()/update() operations get invoked on the aggregate buffer without calling 
initialize
 Key: SPARK-27207
 URL: https://issues.apache.org/jira/browse/SPARK-27207
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.0.0
Reporter: Parth Gandhi


Normally, the aggregate operations that are invoked for an aggregation buffer 
for User Defined Aggregate Functions(UDAF) follow the order like initialize(), 
update(), eval() OR initialize(), merge(), eval(). However, after a certain 
threshold configurable by 
spark.sql.objectHashAggregate.sortBased.fallbackThreshold is reached, 
ObjectHashAggregate falls back to SortBasedAggregator which invokes the merge 
or update operation without calling initialize() on the aggregate buffer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Dongjoon Hyun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796434#comment-16796434
 ] 

Dongjoon Hyun commented on SPARK-27194:
---

Hi, [~ajithshetty]. Just out of curiosity, according to the `Affected 
Versions`, do you observe this in 2.3.1 and 2.3.2? Is it possible to use 2.3.3 
in your cluster since that's already released?

> Job failures when task attempts do not clean up spark-staging parquet files
> ---
>
> Key: SPARK-27194
> URL: https://issues.apache.org/jira/browse/SPARK-27194
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Reza Safi
>Priority: Major
>
> When a container fails for some reason (for example when killed by yarn for 
> exceeding memory limits), the subsequent task attempts for the tasks that 
> were running on that container all fail with a FileAlreadyExistsException. 
> The original task attempt does not seem to successfully call abortTask (or at 
> least its "best effort" delete is unsuccessful) and clean up the parquet file 
> it was writing to, so when later task attempts try to write to the same 
> spark-staging directory using the same file name, the job fails.
> Here is what transpires in the logs:
> The container where task 200.0 is running is killed and the task is lost:
> {code}
> 19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
> t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 
> GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
>  19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 
> 0.0 (TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 
> exited caused by one of the running tasks) Reason: Container killed by YARN 
> for exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider 
> boosting spark.yarn.executor.memoryOverhead.
> {code}
> The task is re-attempted on a different executor and fails because the 
> part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
> already exists:
> {code}
> 19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
> (TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:109)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client 17.161.235.91 already exists
> {code}
> The job fails when the the configured task attempts (spark.task.maxFailures) 
> have failed with the same error:
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 
> in stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 
> 0.0 (TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  ...
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client i.p.a.d already exists
> {code}
> SPARK-26682 wasn't the root cause here, since there wasn't any stage 
> reattempt.
> This issue seems to happen when 
> spark.sql.sources.partitionOverwriteMode=dynamic. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Dongjoon Hyun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-27194:
--
Description: 
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:
{code}
19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.

The task is re-attempted on a different executor and fails because the 
part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
already exists:

19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
(TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:109)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client 17.161.235.91 already exists

The job fails when the the configured task attempts (spark.task.maxFailures) 
have failed with the same error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 in 
stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 0.0 
(TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 ...
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client i.p.a.d already exists
{code}

SPARK-26682 wasn't the root cause here, since there wasn't any stage reattempt.

This issue seems to happen when 
spark.sql.sources.partitionOverwriteMode=dynamic. 

 

  was:
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:

19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.

The task 

[jira] [Updated] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Dongjoon Hyun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-27194:
--
Description: 
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:
{code}
19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 
spark.yarn.executor.memoryOverhead.
{code}

The task is re-attempted on a different executor and fails because the 
part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
already exists:
{code}
19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
(TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:109)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client 17.161.235.91 already exists
{code}

The job fails when the the configured task attempts (spark.task.maxFailures) 
have failed with the same error:
{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 in 
stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 0.0 
(TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
failed while writing rows.
 at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
 ...
 Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
/user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
 for client i.p.a.d already exists
{code}

SPARK-26682 wasn't the root cause here, since there wasn't any stage reattempt.

This issue seems to happen when 
spark.sql.sources.partitionOverwriteMode=dynamic. 

 

  was:
When a container fails for some reason (for example when killed by yarn for 
exceeding memory limits), the subsequent task attempts for the tasks that were 
running on that container all fail with a FileAlreadyExistsException. The 
original task attempt does not seem to successfully call abortTask (or at least 
its "best effort" delete is unsuccessful) and clean up the parquet file it was 
writing to, so when later task attempts try to write to the same spark-staging 
directory using the same file name, the job fails.

Here is what transpires in the logs:

The container where task 200.0 is running is killed and the task is lost:
{code}
19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 GB 
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
 19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 0.0 
(TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 exited 
caused by one of the running tasks) Reason: Container killed by YARN for 
exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider boosting 

[jira] [Resolved] (SPARK-27179) Exclude javax.ws.rs:jsr311-api from hadoop-client

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen resolved SPARK-27179.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 24114
[https://github.com/apache/spark/pull/24114]

> Exclude javax.ws.rs:jsr311-api from hadoop-client
> -
>
> Key: SPARK-27179
> URL: https://issues.apache.org/jira/browse/SPARK-27179
> Project: Spark
>  Issue Type: Sub-task
>  Components: Build
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Minor
> Fix For: 3.0.0
>
>
> Since YARN-7113(Hadoop-3.1.0), hadoop-client add javax.ws.rs:jsr311-api to 
> its dependency . But it conflict with javax.ws.rs-api-2.0.1.jar.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27179) Exclude javax.ws.rs:jsr311-api from hadoop-client

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen reassigned SPARK-27179:
-

Assignee: Yuming Wang
Priority: Minor  (was: Major)

> Exclude javax.ws.rs:jsr311-api from hadoop-client
> -
>
> Key: SPARK-27179
> URL: https://issues.apache.org/jira/browse/SPARK-27179
> Project: Spark
>  Issue Type: Sub-task
>  Components: Build
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Minor
>
> Since YARN-7113(Hadoop-3.1.0), hadoop-client add javax.ws.rs:jsr311-api to 
> its dependency . But it conflict with javax.ws.rs-api-2.0.1.jar.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24668) PySpark crashes when getting the webui url if the webui is disabled

[Karthik Palaniappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796263#comment-16796263
 ] 

Karthik Palaniappan commented on SPARK-24668:
-

Please take it :) I didn't have time to work on it, sadly.

> PySpark crashes when getting the webui url if the webui is disabled
> ---
>
> Key: SPARK-24668
> URL: https://issues.apache.org/jira/browse/SPARK-24668
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.3.0, 2.4.0
> Environment: * Spark 2.3.0
>  * Spark-on-YARN
>  * Java 8
>  * Python 3.6.5
>  * Jupyter 4.4.0
>Reporter: Karthik Palaniappan
>Priority: Minor
>
> Repro:
>  
> Evaluate `sc` in a Jupyter notebook:
>  
>  
> {{---}}
> {{Py4JJavaError                             Traceback (most recent call 
> last)}}
> {{/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in 
> __call__(self, obj)}}
> {{    343             method = get_real_method(obj, self.print_method)}}
> {{    344             if method is not None:}}
> {{--> 345                 return method()}}
> {{    346             return None}}
> {{    347         else:}}
> {{/usr/lib/spark/python/pyspark/context.py in _repr_html_(self)}}
> {{    261         }}
> {{    262         """.format(}}
> {{--> 263             sc=self}}
> {{    264         )}}
> {{    265 }}
> {{/usr/lib/spark/python/pyspark/context.py in uiWebUrl(self)}}
> {{    373     def uiWebUrl(self):}}
> {{    374         """Return the URL of the SparkUI instance started by this 
> SparkContext"""}}
> {{--> 375         return 
> self._[jsc.sc|https://www.google.com/url?q=http://jsc.sc=D=AFQjCNHUwO0Cf3OHs1QafBFXzShZ_PU8IQ]().uiWebUrl().get()}}
> {{    376 }}
> {{    377     @property}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in 
> __call__(self, *args)}}
> {{   1158         answer = self.gateway_client.send_command(command)}}
> {{   1159         return_value = get_return_value(}}
> {{-> 1160             answer, self.gateway_client, self.target_id, 
> [self.name|https://www.google.com/url?q=http://self.name=D=AFQjCNEu_LlQOduOrIyV64UgIuRgm6Ea2w])}}
> {{   1161 }}
> {{   1162         for temp_arg in temp_args:}}
> {{/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
> {{     61     def deco(*a, **kw):}}
> {{     62         try:}}
> {{---> 63             return f(*a, **kw)}}
> {{     64         except py4j.protocol.Py4JJavaError as e:}}
> {{     65             s = e.java_exception.toString()}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in 
> get_return_value(answer, gateway_client, target_id, name)}}
> {{    318                 raise Py4JJavaError(}}
> {{    319                     "An error occurred while calling 
> \{0}{1}\{2}.\n".}}
> {{--> 320                     format(target_id, ".", name), value)}}
> {{    321             else:}}
> {{    322                 raise Py4JError(}}
> {{Py4JJavaError: An error occurred while calling o80.get.}}
> {{: java.util.NoSuchElementException: None.get}}
> {{        at scala.None$.get(Option.scala:347)}}
> {{        at scala.None$.get(Option.scala:345)}}
> {{        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{        at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)}}
> {{        at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)}}
> {{        at py4j.Gateway.invoke(Gateway.java:282)}}
> {{        at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)}}
> {{        at py4j.commands.CallCommand.execute(CallCommand.java:79)}}
> {{        at py4j.GatewayConnection.run(GatewayConnection.java:214)}}
> {{        at java.lang.Thread.run(Thread.java:748)}}
>  
> PySpark only prints out the web ui url in `_repr_html`, not `__repr__`, so 
> this only happens in notebooks that render html, not the pyspark shell. 
> [https://github.com/apache/spark/commit/f654b39a63d4f9b118733733c7ed2a1b58649e3d]
>  
> Disabling Spark's UI with `spark.ui.enabled` *is* valuable outside of tests. 
> A couple reasons that come to mind:
> 1) If you run multiple spark applications from one machine, Spark 
> irritatingly starts picking the same port (4040), as the first application, 
> then increments (4041, 4042, etc) until it finds an open port. If you are 
> running 10 spark apps, then the 11th prints out 10 warnings about ports being 
> taken until it finally finds one.
> 2) You can serve the 

[jira] [Created] (SPARK-27206) Using slice method with streaming api's Interval on DStream

[Aarthi (JIRA)]

Aarthi created SPARK-27206:
--

 Summary: Using slice method with streaming api's Interval on 
DStream
 Key: SPARK-27206
 URL: https://issues.apache.org/jira/browse/SPARK-27206
 Project: Spark
  Issue Type: Question
  Components: DStreams
Affects Versions: 2.4.0
 Environment: Linux, standalone spark
Reporter: Aarthi


Hi. I am in need to slice a DStream that receives data from a custom receiver 
(implemented with Spark's Receiver). There are two options to do this

1. slice(fromTime: 
[Time|http://spark.apache.org/docs/2.3.1/api/scala/org/apache/spark/streaming/Time.html],
 toTime: 
[Time|http://spark.apache.org/docs/2.3.1/api/scala/org/apache/spark/streaming/Time.html])
2. slice(interval: Interval)

Although the second option is a public method, the Interval class is private. 
Can you please help me understand how to use this api with 
org.apache.spark.streaming.Interval ?

Thanks, Aarthi

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27205:


Assignee: Apache Spark

> spark-shell with packages option fails to load transitive dependencies even 
> ivy successfully pulls jars
> ---
>
> Key: SPARK-27205
> URL: https://issues.apache.org/jira/browse/SPARK-27205
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Assignee: Apache Spark
>Priority: Major
>
> I found this bug while testing my patch regarding Spark SQL Kafka module - I 
> tend to open spark-shell and link kafka module via `–packages`.
> When we run
> {code:java}
> ./bin/spark-shell --packages 
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
> we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
> work for current master branch.
> There's not enough evidence as well as I have no idea what's happening here 
> even with `–verbose` option, so I had to spend couple of hours dealing with 
> git bisect.
> Turned out the commit introducing the bug was SPARK-26977 
> ([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27205:


Assignee: (was: Apache Spark)

> spark-shell with packages option fails to load transitive dependencies even 
> ivy successfully pulls jars
> ---
>
> Key: SPARK-27205
> URL: https://issues.apache.org/jira/browse/SPARK-27205
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
>
> I found this bug while testing my patch regarding Spark SQL Kafka module - I 
> tend to open spark-shell and link kafka module via `–packages`.
> When we run
> {code:java}
> ./bin/spark-shell --packages 
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
> we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
> work for current master branch.
> There's not enough evidence as well as I have no idea what's happening here 
> even with `–verbose` option, so I had to spend couple of hours dealing with 
> git bisect.
> Turned out the commit introducing the bug was SPARK-26977 
> ([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-27205:
-
Description: 
I found this bug while testing my patch regarding Spark SQL Kafka module - I 
tend to open spark-shell and link kafka module via `–packages`.

When we run
{code:java}
./bin/spark-shell --packages 
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
work for current master branch.

There's not enough evidence as well as I have no idea what's happening here 
even with `–verbose` option, so I had to spend couple of hours dealing with git 
bisect.

Turned out the commit introducing the bug was SPARK-26977 
([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).

 

  was:
I found this bug while testing my patch regarding Spark SQL Kafka module - I 
tend to open spark-shell and link kafka module via `–packages`.

When we run
{code:java}
./bin/spark-shell --packages 
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
work for current master branch.

There's not enough evidence as well as I have no idea what's happening here 
even with `–verbose` option, so I had to spend couple of hours dealing with git 
bisect.

Turned out the commit introducing the bug was 
[81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51].

 


> spark-shell with packages option fails to load transitive dependencies even 
> ivy successfully pulls jars
> ---
>
> Key: SPARK-27205
> URL: https://issues.apache.org/jira/browse/SPARK-27205
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
>
> I found this bug while testing my patch regarding Spark SQL Kafka module - I 
> tend to open spark-shell and link kafka module via `–packages`.
> When we run
> {code:java}
> ./bin/spark-shell --packages 
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
> we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
> work for current master branch.
> There's not enough evidence as well as I have no idea what's happening here 
> even with `–verbose` option, so I had to spend couple of hours dealing with 
> git bisect.
> Turned out the commit introducing the bug was SPARK-26977 
> ([81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51]).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

[Jungtaek Lim (JIRA)]

Jungtaek Lim created SPARK-27205:


 Summary: spark-shell with packages option fails to load transitive 
dependencies even ivy successfully pulls jars
 Key: SPARK-27205
 URL: https://issues.apache.org/jira/browse/SPARK-27205
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 3.0.0
Reporter: Jungtaek Lim


I found this bug while testing my patch regarding Spark SQL Kafka module - I 
tend to open spark-shell and link kafka module via `–packages`.

When we run
{code:java}
./bin/spark-shell --packages 
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
work for current master branch.

There's not enough evidence as well as I have no idea what's happening here 
even with `–verbose` option, so I had to spend couple of hours dealing with git 
bisect.

Turned out the commit introducing the bug was 
[81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51].

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796125#comment-16796125
 ] 

Jungtaek Lim commented on SPARK-27205:
--

I have a patch now: it's not just a rollback of commit. Will submit it shortly.

> spark-shell with packages option fails to load transitive dependencies even 
> ivy successfully pulls jars
> ---
>
> Key: SPARK-27205
> URL: https://issues.apache.org/jira/browse/SPARK-27205
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
>
> I found this bug while testing my patch regarding Spark SQL Kafka module - I 
> tend to open spark-shell and link kafka module via `–packages`.
> When we run
> {code:java}
> ./bin/spark-shell --packages 
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0{code}
> we should be able to import "org.apache.kafka" in spark-shell, but it doesn't 
> work for current master branch.
> There's not enough evidence as well as I have no idea what's happening here 
> even with `–verbose` option, so I had to spend couple of hours dealing with 
> git bisect.
> Turned out the commit introducing the bug was 
> [81dd21fda99da48ed76adb739a07d1dabf1ffb51|https://github.com/apache/spark/commit/81dd21fda99da48ed76adb739a07d1dabf1ffb51].
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27204) First time Loading application page from History Server is taking time when event log size is huge

[ABHISHEK KUMAR GUPTA (JIRA)]

ABHISHEK KUMAR GUPTA created SPARK-27204:


 Summary: First time Loading application page from History Server 
is taking time when event log size is huge
 Key: SPARK-27204
 URL: https://issues.apache.org/jira/browse/SPARK-27204
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core, Web UI
Affects Versions: 2.4.0, 2.3.3
Reporter: ABHISHEK KUMAR GUPTA


1. Launch spark shell and submit a long running job.
2. Measure the loading time of Job History Page first time.
3. For Example Event Log Size = 18GB, With disk store, Application page Loading 
time takes first time 47 Min



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27204) First time Loading application page from History Server is taking time when event log size is huge

[shahid (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796110#comment-16796110
 ] 

shahid commented on SPARK-27204:


Thanks. I will analyze,

> First time Loading application page from History Server is taking time when 
> event log size is huge
> --
>
> Key: SPARK-27204
> URL: https://issues.apache.org/jira/browse/SPARK-27204
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, Web UI
>Affects Versions: 2.3.3, 2.4.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> 1. Launch spark shell and submit a long running job.
> 2. Measure the loading time of Job History Page first time.
> 3. For Example Event Log Size = 18GB, With disk store, Application page 
> Loading time takes first time 47 Min



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27203) Spark Fails to read a view using CTE (WITH clause) and created via beeline

[Igor Ngouagna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igor Ngouagna updated SPARK-27203:
--
Environment: (was: *Spark: 2.1.1*

*Beeline: 1.2.1000*)

> Spark Fails to read a view using CTE (WITH clause) and created via beeline 
> ---
>
> Key: SPARK-27203
> URL: https://issues.apache.org/jira/browse/SPARK-27203
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Igor Ngouagna
>Priority: Major
>
> Spark fails when trying to read a view which code involve CTE, and which is 
> created via beeline.
> For example, considering the following view, created via Beeline:
> {code:sql}
> create view db.test as 
> with q1 as (select 1 as n)
> select n from q1
> {code}
> When you do
> {code:java}
> spark.sql("select * from db.test").show()
> {code}
> The output is like
> {code}
> 'Table or view not found: q1; line 2 pos 14'
> Traceback (most recent call last):
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/session.py",
>  line 545, in sql
> return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/py4j-0.10.4-src.zip/py4j/java_gateway.py",
>  line 1133, in __call__
> answer, self.gateway_client, self.target_id, self.name)
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/utils.py",
>  line 69, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.AnalysisException: 'Table or view not found: q1; line 2 pos 
> 14'
> {code}
>  
> *Spark: 2.1.1*
> *Beeline: 1.2.1000*
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27203) Spark Fails to read a view using CTE (WITH clause) and created via beeline

[Igor Ngouagna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igor Ngouagna updated SPARK-27203:
--
Summary: Spark Fails to read a view using CTE (WITH clause) and created via 
beeline   (was: Fails to read a view using CTE (WITH clause) and created via 
beeline )

> Spark Fails to read a view using CTE (WITH clause) and created via beeline 
> ---
>
> Key: SPARK-27203
> URL: https://issues.apache.org/jira/browse/SPARK-27203
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.1
> Environment: *Spark: 2.1.1*
> *Beeline: 1.2.1000*
>Reporter: Igor Ngouagna
>Priority: Major
>
> Spark fails when trying to read a view which code involve CTE, and which is 
> created via beeline.
> For example, considering the following view, created via Beeline:
> {code:sql}
> create view db.test as 
> with q1 as (select 1 as n)
> select n from q1
> {code}
> When you do
> {code:java}
> spark.sql("select * from db.test").show()
> {code}
> The output is like
> {code}
> 'Table or view not found: q1; line 2 pos 14'
> Traceback (most recent call last):
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/session.py",
>  line 545, in sql
> return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/py4j-0.10.4-src.zip/py4j/java_gateway.py",
>  line 1133, in __call__
> answer, self.gateway_client, self.target_id, self.name)
>   File 
> "/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/utils.py",
>  line 69, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.AnalysisException: 'Table or view not found: q1; line 2 pos 
> 14'
> {code}
>  
> *Spark: 2.1.1*
> *Beeline: 1.2.1000*
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27203) Fails to read a view using CTE (WITH clause) and created via beeline

[Igor Ngouagna (JIRA)]

Igor Ngouagna created SPARK-27203:
-

 Summary: Fails to read a view using CTE (WITH clause) and created 
via beeline 
 Key: SPARK-27203
 URL: https://issues.apache.org/jira/browse/SPARK-27203
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.1.1
 Environment: *Spark: 2.1.1*

*Beeline: 1.2.1000*
Reporter: Igor Ngouagna


Spark fails when trying to read a view which code involve CTE, and which is 
created via beeline.

For example, considering the following view, created via Beeline:
{code:sql}
create view db.test as 
with q1 as (select 1 as n)
select n from q1
{code}
When you do
{code:java}
spark.sql("select * from db.test").show()
{code}
The output is like
{code}
'Table or view not found: q1; line 2 pos 14'
Traceback (most recent call last):
  File 
"/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/session.py",
 line 545, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File 
"/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/py4j-0.10.4-src.zip/py4j/java_gateway.py",
 line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
  File 
"/DATA/fs11/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552973526615_3878/container_e380_1552973526615_3878_01_01/pyspark.zip/pyspark/sql/utils.py",
 line 69, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: 'Table or view not found: q1; line 2 pos 
14'
{code}
 

*Spark: 2.1.1*

*Beeline: 1.2.1000*

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27202) update comments to keep according with code

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27202:


Assignee: (was: Apache Spark)

> update comments to keep according with code
> ---
>
> Key: SPARK-27202
> URL: https://issues.apache.org/jira/browse/SPARK-27202
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: EdisonWang
>Priority: Minor
>
> update comments in InMemoryFileIndex.scala to keep according with code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27202) update comments to keep according with code

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27202:


Assignee: Apache Spark

> update comments to keep according with code
> ---
>
> Key: SPARK-27202
> URL: https://issues.apache.org/jira/browse/SPARK-27202
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: EdisonWang
>Assignee: Apache Spark
>Priority: Minor
>
> update comments in InMemoryFileIndex.scala to keep according with code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27202) update comments to keep according with code

[EdisonWang (JIRA)]

EdisonWang created SPARK-27202:
--

 Summary: update comments to keep according with code
 Key: SPARK-27202
 URL: https://issues.apache.org/jira/browse/SPARK-27202
 Project: Spark
  Issue Type: Improvement
  Components: SQL
Affects Versions: 2.4.0
Reporter: EdisonWang


update comments in InMemoryFileIndex.scala to keep according with code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27201) Show full job description on click

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27201:


Assignee: (was: Apache Spark)

> Show full job description on click
> --
>
> Key: SPARK-27201
> URL: https://issues.apache.org/jira/browse/SPARK-27201
> Project: Spark
>  Issue Type: Task
>  Components: Web UI
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> Previously, in https://github.com/apache/spark/pull/6646 there was an 
> improvement to show full job description after double click.
> I think this is a bit hard to be noticed by some users. I suggest changing 
> the event to one click.
> Also, after the full description is shown, another click should be able to 
> hide the overflow text again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27201) Show full job description on click

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27201:


Assignee: Apache Spark

> Show full job description on click
> --
>
> Key: SPARK-27201
> URL: https://issues.apache.org/jira/browse/SPARK-27201
> Project: Spark
>  Issue Type: Task
>  Components: Web UI
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Assignee: Apache Spark
>Priority: Major
>
> Previously, in https://github.com/apache/spark/pull/6646 there was an 
> improvement to show full job description after double click.
> I think this is a bit hard to be noticed by some users. I suggest changing 
> the event to one click.
> Also, after the full description is shown, another click should be able to 
> hide the overflow text again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27201) Show full job description on click

[Gengliang Wang (JIRA)]

Gengliang Wang created SPARK-27201:
--

 Summary: Show full job description on click
 Key: SPARK-27201
 URL: https://issues.apache.org/jira/browse/SPARK-27201
 Project: Spark
  Issue Type: Task
  Components: Web UI
Affects Versions: 3.0.0
Reporter: Gengliang Wang


Previously, in https://github.com/apache/spark/pull/6646 there was an 
improvement to show full job description after double click.
I think this is a bit hard to be noticed by some users. I suggest changing the 
event to one click.
Also, after the full description is shown, another click should be able to hide 
the overflow text again.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26739) Standardized Join Types for DataFrames

[Valeria Vasylieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796045#comment-16796045
 ] 

Valeria Vasylieva commented on SPARK-26739:
---

[~slehan] hi! I strongly agree this change will make Dataset/DataFrame API 
users much happier, me at least. 

I would like to work on this task.

 

There are several questions though:

1) Will this change be accepted, if implemented?

2) If yes, do we need to mark methods using strings as deprecated?

 

[~hyukjin.kwon], [~r...@databricks.com], [~cloud_fan] could one of you please 
answer these questions?

Thank you!

> Standardized Join Types for DataFrames
> --
>
> Key: SPARK-26739
> URL: https://issues.apache.org/jira/browse/SPARK-26739
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Skyler Lehan
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> h3. *Q1.* What are you trying to do? Articulate your objectives using 
> absolutely no jargon.
> Currently, in the join functions on 
> [DataFrames|http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset],
>  the join types are defined via a string parameter called joinType. In order 
> for a developer to know which joins are possible, they must look up the API 
> call for join. While this works fine, it can cause the developer to make a 
> typo resulting in improper joins and/or unexpected errors that aren't evident 
> at compile time. The objective of this improvement would be to allow 
> developers to use a common definition for join types (by enum or constants) 
> called JoinTypes. This would contain the possible joins and remove the 
> possibility of a typo. It would also allow Spark to alter the names of the 
> joins in the future without impacting end-users.
> h3. *Q2.* What problem is this proposal NOT designed to solve?
> The problem this solves is extremely narrow, it would not solve anything 
> other than providing a common definition for join types.
> h3. *Q3.* How is it done today, and what are the limits of current practice?
> Currently, developers must join two DataFrames like so:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> "left_outer")
> {code}
> Where they manually type the join type. As stated above, this:
>  * Requires developers to manually type in the join
>  * Can cause possibility of typos
>  * Restricts renaming of join types as its a literal string
>  * Does not restrict and/or compile check the join type being used, leading 
> to runtime errors
> h3. *Q4.* What is new in your approach and why do you think it will be 
> successful?
> The new approach would use constants or *more preferably an enum*, something 
> like this:
> {code:java}
> val resultDF = leftDF.join(rightDF, col("ID") === col("RightID"), 
> JoinType.LEFT_OUTER)
> {code}
> This would provide:
>  * In code reference/definitions of the possible join types
>  ** This subsequently allows the addition of scaladoc of what each join type 
> does and how it operates
>  * Removes possibilities of a typo on the join type
>  * Provides compile time checking of the join type (only if an enum is used)
> To clarify, if JoinType is a constant, it would just fill in the joinType 
> string parameter for users. If an enum is used, it would restrict the domain 
> of possible join types to whatever is defined in the future JoinType enum. 
> The enum is preferred, however it would take longer to implement.
> h3. *Q5.* Who cares? If you are successful, what difference will it make?
> Developers using Apache Spark will care. This will make the join function 
> easier to wield and lead to less runtime errors. It will save time by 
> bringing join type validation at compile time. It will also provide in code 
> reference to the join types, which saves the developer time of having to look 
> up and navigate the multiple join functions to find the possible join types. 
> In addition to that, the resulting constants/enum would have documentation on 
> how that join type works.
> h3. *Q6.* What are the risks?
> Users of Apache Spark who currently use strings to define their join types 
> could be impacted if an enum is chosen as the common definition. This risk 
> can be mitigated by using string constants. The string constants would be the 
> exact same string as the string literals used today. For example:
> {code:java}
> JoinType.INNER = "inner"
> {code}
> If an enum is still the preferred way of defining the join types, new join 
> functions could be added that take in these enums and the join calls that 
> contain string parameters for joinType could be deprecated. This would give 
> developers a chance to change over to the new join types.
> h3. *Q7.* How long will it take?
> A few 

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796039#comment-16796039
 ] 

acupple commented on SPARK-27169:
-

The full event log is too large.

unknown stage log：

[^stage_3511.log]

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, 
> job_1924.log, stage_3511.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

acupple updated SPARK-27169:

Attachment: stage_3511.log

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, 
> job_1924.log, stage_3511.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26176) Verify column name when creating table via `STORED AS`

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan resolved SPARK-26176.
-
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 24075
[https://github.com/apache/spark/pull/24075]

> Verify column name when creating table via `STORED AS`
> --
>
> Key: SPARK-26176
> URL: https://issues.apache.org/jira/browse/SPARK-26176
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Xiao Li
>Assignee: Sujith Chacko
>Priority: Minor
>  Labels: starter
> Fix For: 3.0.0
>
>
> We can issue a reasonable exception when we creating Parquet native tables, 
> {code:java}
> CREATE TABLE TAB1TEST USING PARQUET AS SELECT COUNT(ID) FROM TAB1;
> {code}
> {code:java}
> org.apache.spark.sql.AnalysisException: Attribute name "count(ID)" contains 
> invalid character(s) among " ,;{}()\n\t=". Please use alias to rename it.;
> {code}
> However, the error messages are misleading when we create a table using the 
> Hive serde "STORED AS"
> {code:java}
> CREATE TABLE TAB1TEST STORED AS PARQUET AS SELECT COUNT(ID) FROM TAB1;
> {code}
> {code:java}
> 18/11/26 09:04:44 ERROR SparkSQLDriver: Failed in [CREATE TABLE TAB2TEST 
> stored as parquet AS SELECT COUNT(col1) FROM TAB1]
> org.apache.spark.SparkException: Job aborted.
>   at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:196)
>   at 
> org.apache.spark.sql.hive.execution.SaveAsHiveFile.saveAsHiveFile(SaveAsHiveFile.scala:97)
>   at 
> org.apache.spark.sql.hive.execution.SaveAsHiveFile.saveAsHiveFile$(SaveAsHiveFile.scala:48)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile(InsertIntoHiveTable.scala:66)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:201)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
>   at 
> org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:86)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:113)
>   at 
> org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:201)
>   at 
> org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3270)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:147)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
>   at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3266)
>   at org.apache.spark.sql.Dataset.(Dataset.scala:201)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:86)
>   at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:655)
>   at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:685)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:371)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:274)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>   at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:852)
>   at 
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>   at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>   at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>   at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936)
>   at 

[jira] [Assigned] (SPARK-26176) Verify column name when creating table via `STORED AS`

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan reassigned SPARK-26176:
---

Assignee: Sujith Chacko

> Verify column name when creating table via `STORED AS`
> --
>
> Key: SPARK-26176
> URL: https://issues.apache.org/jira/browse/SPARK-26176
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Xiao Li
>Assignee: Sujith Chacko
>Priority: Minor
>  Labels: starter
>
> We can issue a reasonable exception when we creating Parquet native tables, 
> {code:java}
> CREATE TABLE TAB1TEST USING PARQUET AS SELECT COUNT(ID) FROM TAB1;
> {code}
> {code:java}
> org.apache.spark.sql.AnalysisException: Attribute name "count(ID)" contains 
> invalid character(s) among " ,;{}()\n\t=". Please use alias to rename it.;
> {code}
> However, the error messages are misleading when we create a table using the 
> Hive serde "STORED AS"
> {code:java}
> CREATE TABLE TAB1TEST STORED AS PARQUET AS SELECT COUNT(ID) FROM TAB1;
> {code}
> {code:java}
> 18/11/26 09:04:44 ERROR SparkSQLDriver: Failed in [CREATE TABLE TAB2TEST 
> stored as parquet AS SELECT COUNT(col1) FROM TAB1]
> org.apache.spark.SparkException: Job aborted.
>   at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:196)
>   at 
> org.apache.spark.sql.hive.execution.SaveAsHiveFile.saveAsHiveFile(SaveAsHiveFile.scala:97)
>   at 
> org.apache.spark.sql.hive.execution.SaveAsHiveFile.saveAsHiveFile$(SaveAsHiveFile.scala:48)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile(InsertIntoHiveTable.scala:66)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:201)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
>   at 
> org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:86)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
>   at 
> org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:113)
>   at 
> org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:201)
>   at 
> org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3270)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:147)
>   at 
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
>   at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3266)
>   at org.apache.spark.sql.Dataset.(Dataset.scala:201)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:86)
>   at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:655)
>   at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:685)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:371)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:274)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>   at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:852)
>   at 
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>   at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>   at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>   at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in 

[jira] [Resolved] (SPARK-27151) ClearCacheCommand extends IgnoreCahedData to avoid plan node copys

[Takeshi Yamamuro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeshi Yamamuro resolved SPARK-27151.
--
   Resolution: Fixed
 Assignee: Takeshi Yamamuro
Fix Version/s: 3.0.0

Resolved by https://github.com/apache/spark/pull/24081

> ClearCacheCommand extends IgnoreCahedData to avoid plan node copys
> --
>
> Key: SPARK-27151
> URL: https://issues.apache.org/jira/browse/SPARK-27151
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Takeshi Yamamuro
>Assignee: Takeshi Yamamuro
>Priority: Trivial
> Fix For: 3.0.0
>
>
> In SPARK-27011, we introduced IgnoreCahedData to avoid plan node copys in 
> CacheManager.
> Since ClearCacheCommand has no argument, it also can extend IgnoreCahedData.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:19 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:32 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true?

Because the above fix however, doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

*ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

*ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:24 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

*ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

*ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when **ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:24 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

*ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when **ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when **ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:17 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java*_;
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan reassigned SPARK-26555:
---

Assignee: Martin Loncaric

> Thread safety issue causes createDataset to fail with misleading errors
> ---
>
> Key: SPARK-26555
> URL: https://issues.apache.org/jira/browse/SPARK-26555
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Martin Loncaric
>Assignee: Martin Loncaric
>Priority: Major
>
> This can be replicated (~2% of the time) with
> {code:scala}
> import java.sql.Timestamp
> import java.util.concurrent.{Executors, Future}
> import org.apache.spark.sql.SparkSession
> import scala.collection.mutable.ListBuffer
> import scala.concurrent.ExecutionContext
> import scala.util.Random
> object Main {
>   def main(args: Array[String]): Unit = {
> val sparkSession = SparkSession.builder
>   .getOrCreate()
> import sparkSession.implicits._
> val executor = Executors.newFixedThreadPool(1)
> try {
>   implicit val xc: ExecutionContext = 
> ExecutionContext.fromExecutorService(executor)
>   val futures = new ListBuffer[Future[_]]()
>   for (i <- 1 to 3) {
> futures += executor.submit(new Runnable {
>   override def run(): Unit = {
> val d = if (Random.nextInt(2) == 0) Some("d value") else None
> val e = if (Random.nextInt(2) == 0) Some(5.0) else None
> val f = if (Random.nextInt(2) == 0) Some(6.0) else None
> println("DEBUG", d, e, f)
> sparkSession.createDataset(Seq(
>   MyClass(new Timestamp(1L), "b", "c", d, e, f)
> ))
>   }
> })
>   }
>   futures.foreach(_.get())
> } finally {
>   println("SHUTDOWN")
>   executor.shutdown()
>   sparkSession.stop()
> }
>   }
>   case class MyClass(
> a: Timestamp,
> b: String,
> c: String,
> d: Option[String],
> e: Option[Double],
> f: Option[Double]
>   )
> }
> {code}
> So it will usually come up during
> {code:bash}
> for i in $(seq 1 200); do
>   echo $i
>   spark-submit --master local[4] target/scala-2.11/spark-test_2.11-0.1.jar
> done
> {code}
> causing a variety of possible errors, such as
> {code}Exception in thread "main" java.util.concurrent.ExecutionException: 
> scala.MatchError: scala.Option[String] (of class 
> scala.reflect.internal.Types$ClassArgsTypeRef)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> Caused by: scala.MatchError: scala.Option[String] (of class 
> scala.reflect.internal.Types$ClassArgsTypeRef)
>   at 
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$deserializerFor$1.apply(ScalaReflection.scala:210){code}
> or
> {code}Exception in thread "main" java.util.concurrent.ExecutionException: 
> java.lang.UnsupportedOperationException: Schema for type 
> scala.Option[scala.Double] is not supported
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> Caused by: java.lang.UnsupportedOperationException: Schema for type 
> scala.Option[scala.Double] is not supported
>   at 
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:789){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan resolved SPARK-26555.
-
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 24085
[https://github.com/apache/spark/pull/24085]

> Thread safety issue causes createDataset to fail with misleading errors
> ---
>
> Key: SPARK-26555
> URL: https://issues.apache.org/jira/browse/SPARK-26555
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Martin Loncaric
>Assignee: Martin Loncaric
>Priority: Major
> Fix For: 3.0.0
>
>
> This can be replicated (~2% of the time) with
> {code:scala}
> import java.sql.Timestamp
> import java.util.concurrent.{Executors, Future}
> import org.apache.spark.sql.SparkSession
> import scala.collection.mutable.ListBuffer
> import scala.concurrent.ExecutionContext
> import scala.util.Random
> object Main {
>   def main(args: Array[String]): Unit = {
> val sparkSession = SparkSession.builder
>   .getOrCreate()
> import sparkSession.implicits._
> val executor = Executors.newFixedThreadPool(1)
> try {
>   implicit val xc: ExecutionContext = 
> ExecutionContext.fromExecutorService(executor)
>   val futures = new ListBuffer[Future[_]]()
>   for (i <- 1 to 3) {
> futures += executor.submit(new Runnable {
>   override def run(): Unit = {
> val d = if (Random.nextInt(2) == 0) Some("d value") else None
> val e = if (Random.nextInt(2) == 0) Some(5.0) else None
> val f = if (Random.nextInt(2) == 0) Some(6.0) else None
> println("DEBUG", d, e, f)
> sparkSession.createDataset(Seq(
>   MyClass(new Timestamp(1L), "b", "c", d, e, f)
> ))
>   }
> })
>   }
>   futures.foreach(_.get())
> } finally {
>   println("SHUTDOWN")
>   executor.shutdown()
>   sparkSession.stop()
> }
>   }
>   case class MyClass(
> a: Timestamp,
> b: String,
> c: String,
> d: Option[String],
> e: Option[Double],
> f: Option[Double]
>   )
> }
> {code}
> So it will usually come up during
> {code:bash}
> for i in $(seq 1 200); do
>   echo $i
>   spark-submit --master local[4] target/scala-2.11/spark-test_2.11-0.1.jar
> done
> {code}
> causing a variety of possible errors, such as
> {code}Exception in thread "main" java.util.concurrent.ExecutionException: 
> scala.MatchError: scala.Option[String] (of class 
> scala.reflect.internal.Types$ClassArgsTypeRef)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> Caused by: scala.MatchError: scala.Option[String] (of class 
> scala.reflect.internal.Types$ClassArgsTypeRef)
>   at 
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$deserializerFor$1.apply(ScalaReflection.scala:210){code}
> or
> {code}Exception in thread "main" java.util.concurrent.ExecutionException: 
> java.lang.UnsupportedOperationException: Schema for type 
> scala.Option[scala.Double] is not supported
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> Caused by: java.lang.UnsupportedOperationException: Schema for type 
> scala.Option[scala.Double] is not supported
>   at 
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:789){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:19 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however, doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN and *DataFrameTungsten/InMemoryColumnarQuerySuite* 
passes only when **ByteOrder  is set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:19 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix however doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

The above fix doesn't work on all the test cases and the behavior of 
*ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment 
each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:18 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java]*_
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*[OnHeapColumnVector.java|https://github.com/apache/spark/blob/v2.3.2/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java*_;
 to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade commented on SPARK-26985:
---

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.LITTLE_ENDIAN* the tests passes. 
Because the data is read properly. However in that case some tests of Paraquet 
Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:16 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.BIG_ENDIAN* the tests passes.

Because the float and double data is read properly. However in that case some 
tests of Paraquet Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 *ParquetIOSuite*  passes only when  ByteOrder  is set to 
ByteOrder.LITTLE_ENDIAN 

and  

*DataFrameTungsten/InMemoryColumnarQuerySuite* passes only when **ByteOrder  is 
set to ByteOrder.BIG_ENDIAN.

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.BIG_ENDIAN* the tests passes. 
Because the data is read properly. However in that case some tests of Paraquet 
Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

[Anuja Jakhade (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795931#comment-16795931
 ] 

Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:12 AM:
-

Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.BIG_ENDIAN* the tests passes. 
Because the data is read properly. However in that case some tests of Paraquet 
Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 


was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]

I have observed that after changing the ByteOrder in 
"_*OnHeapColumnVector.java*_" to  *ByteOrder.LITTLE_ENDIAN* the tests passes. 
Because the data is read properly. However in that case some tests of Paraquet 
Module fails. e.x: *ParquetIOSuite.*

Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN 
even when the bigEndianPlatform is true.

Because in that case the fix doesn't work on all the test cases and the 
behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* 
compliment each other. 

 

> Test "access only some column of the all of columns " fails on big endian
> -
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04 
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed 
> References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>  
>Reporter: Anuja Jakhade
>Priority: Major
>  Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, 
> InMemoryColumnarQuerySuite.txt, access only some column of the all of 
> columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am 
> observing test failures for 2 Suites of Project SQL.
>  1. InMemoryColumnarQuerySuite
>  2. DataFrameTungstenSuite
>  In both the cases test "access only some column of the all of columns" fails 
> due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please 
> find attached the log with the details. 
> cache() works perfectly fine if double and  float values are not in picture.
> Inside test !!- access only some column of the all of columns *** FAILED 
> ***



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27200) History Environment tab must sort Configurations/Properties by default

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27200:


Assignee: (was: Apache Spark)

> History Environment tab must sort Configurations/Properties by default
> --
>
> Key: SPARK-27200
> URL: https://issues.apache.org/jira/browse/SPARK-27200
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 3.0.0
>Reporter: Ajith S
>Priority: Minor
>
> Environment Page in SparkUI have all the configuration sorted by key. But 
> this is not the case in History server case, to keep UX same, we can have it 
> sorted in history server too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27200) History Environment tab must sort Configurations/Properties by default

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27200:


Assignee: Apache Spark

> History Environment tab must sort Configurations/Properties by default
> --
>
> Key: SPARK-27200
> URL: https://issues.apache.org/jira/browse/SPARK-27200
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 3.0.0
>Reporter: Ajith S
>Assignee: Apache Spark
>Priority: Minor
>
> Environment Page in SparkUI have all the configuration sorted by key. But 
> this is not the case in History server case, to keep UX same, we can have it 
> sorted in history server too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27200) History Environment tab must sort Configurations/Properties by default

[Ajith S (JIRA)]

Ajith S created SPARK-27200:
---

 Summary: History Environment tab must sort 
Configurations/Properties by default
 Key: SPARK-27200
 URL: https://issues.apache.org/jira/browse/SPARK-27200
 Project: Spark
  Issue Type: Bug
  Components: Web UI
Affects Versions: 3.0.0
Reporter: Ajith S


Environment Page in SparkUI have all the configuration sorted by key. But this 
is not the case in History server case, to keep UX same, we can have it sorted 
in history server too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24668) PySpark crashes when getting the webui url if the webui is disabled

[Hyukjin Kwon (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795903#comment-16795903
 ] 

Hyukjin Kwon commented on SPARK-24668:
--

Please go ahead.

> PySpark crashes when getting the webui url if the webui is disabled
> ---
>
> Key: SPARK-24668
> URL: https://issues.apache.org/jira/browse/SPARK-24668
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.3.0, 2.4.0
> Environment: * Spark 2.3.0
>  * Spark-on-YARN
>  * Java 8
>  * Python 3.6.5
>  * Jupyter 4.4.0
>Reporter: Karthik Palaniappan
>Priority: Minor
>
> Repro:
>  
> Evaluate `sc` in a Jupyter notebook:
>  
>  
> {{---}}
> {{Py4JJavaError                             Traceback (most recent call 
> last)}}
> {{/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in 
> __call__(self, obj)}}
> {{    343             method = get_real_method(obj, self.print_method)}}
> {{    344             if method is not None:}}
> {{--> 345                 return method()}}
> {{    346             return None}}
> {{    347         else:}}
> {{/usr/lib/spark/python/pyspark/context.py in _repr_html_(self)}}
> {{    261         }}
> {{    262         """.format(}}
> {{--> 263             sc=self}}
> {{    264         )}}
> {{    265 }}
> {{/usr/lib/spark/python/pyspark/context.py in uiWebUrl(self)}}
> {{    373     def uiWebUrl(self):}}
> {{    374         """Return the URL of the SparkUI instance started by this 
> SparkContext"""}}
> {{--> 375         return 
> self._[jsc.sc|https://www.google.com/url?q=http://jsc.sc=D=AFQjCNHUwO0Cf3OHs1QafBFXzShZ_PU8IQ]().uiWebUrl().get()}}
> {{    376 }}
> {{    377     @property}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in 
> __call__(self, *args)}}
> {{   1158         answer = self.gateway_client.send_command(command)}}
> {{   1159         return_value = get_return_value(}}
> {{-> 1160             answer, self.gateway_client, self.target_id, 
> [self.name|https://www.google.com/url?q=http://self.name=D=AFQjCNEu_LlQOduOrIyV64UgIuRgm6Ea2w])}}
> {{   1161 }}
> {{   1162         for temp_arg in temp_args:}}
> {{/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
> {{     61     def deco(*a, **kw):}}
> {{     62         try:}}
> {{---> 63             return f(*a, **kw)}}
> {{     64         except py4j.protocol.Py4JJavaError as e:}}
> {{     65             s = e.java_exception.toString()}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in 
> get_return_value(answer, gateway_client, target_id, name)}}
> {{    318                 raise Py4JJavaError(}}
> {{    319                     "An error occurred while calling 
> \{0}{1}\{2}.\n".}}
> {{--> 320                     format(target_id, ".", name), value)}}
> {{    321             else:}}
> {{    322                 raise Py4JError(}}
> {{Py4JJavaError: An error occurred while calling o80.get.}}
> {{: java.util.NoSuchElementException: None.get}}
> {{        at scala.None$.get(Option.scala:347)}}
> {{        at scala.None$.get(Option.scala:345)}}
> {{        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{        at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)}}
> {{        at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)}}
> {{        at py4j.Gateway.invoke(Gateway.java:282)}}
> {{        at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)}}
> {{        at py4j.commands.CallCommand.execute(CallCommand.java:79)}}
> {{        at py4j.GatewayConnection.run(GatewayConnection.java:214)}}
> {{        at java.lang.Thread.run(Thread.java:748)}}
>  
> PySpark only prints out the web ui url in `_repr_html`, not `__repr__`, so 
> this only happens in notebooks that render html, not the pyspark shell. 
> [https://github.com/apache/spark/commit/f654b39a63d4f9b118733733c7ed2a1b58649e3d]
>  
> Disabling Spark's UI with `spark.ui.enabled` *is* valuable outside of tests. 
> A couple reasons that come to mind:
> 1) If you run multiple spark applications from one machine, Spark 
> irritatingly starts picking the same port (4040), as the first application, 
> then increments (4041, 4042, etc) until it finds an open port. If you are 
> running 10 spark apps, then the 11th prints out 10 warnings about ports being 
> taken until it finally finds one.
> 2) You can serve the spark web ui from a dedicated spark history server 
> 

[jira] [Commented] (SPARK-24668) PySpark crashes when getting the webui url if the webui is disabled

[Hyukjin Kwon (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795908#comment-16795908
 ] 

Hyukjin Kwon commented on SPARK-24668:
--

It doesn't usually and neccesarily assign someone. If you open a PR, it will 
automatically assign properly.

> PySpark crashes when getting the webui url if the webui is disabled
> ---
>
> Key: SPARK-24668
> URL: https://issues.apache.org/jira/browse/SPARK-24668
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.3.0, 2.4.0
> Environment: * Spark 2.3.0
>  * Spark-on-YARN
>  * Java 8
>  * Python 3.6.5
>  * Jupyter 4.4.0
>Reporter: Karthik Palaniappan
>Priority: Minor
>
> Repro:
>  
> Evaluate `sc` in a Jupyter notebook:
>  
>  
> {{---}}
> {{Py4JJavaError                             Traceback (most recent call 
> last)}}
> {{/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in 
> __call__(self, obj)}}
> {{    343             method = get_real_method(obj, self.print_method)}}
> {{    344             if method is not None:}}
> {{--> 345                 return method()}}
> {{    346             return None}}
> {{    347         else:}}
> {{/usr/lib/spark/python/pyspark/context.py in _repr_html_(self)}}
> {{    261         }}
> {{    262         """.format(}}
> {{--> 263             sc=self}}
> {{    264         )}}
> {{    265 }}
> {{/usr/lib/spark/python/pyspark/context.py in uiWebUrl(self)}}
> {{    373     def uiWebUrl(self):}}
> {{    374         """Return the URL of the SparkUI instance started by this 
> SparkContext"""}}
> {{--> 375         return 
> self._[jsc.sc|https://www.google.com/url?q=http://jsc.sc=D=AFQjCNHUwO0Cf3OHs1QafBFXzShZ_PU8IQ]().uiWebUrl().get()}}
> {{    376 }}
> {{    377     @property}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in 
> __call__(self, *args)}}
> {{   1158         answer = self.gateway_client.send_command(command)}}
> {{   1159         return_value = get_return_value(}}
> {{-> 1160             answer, self.gateway_client, self.target_id, 
> [self.name|https://www.google.com/url?q=http://self.name=D=AFQjCNEu_LlQOduOrIyV64UgIuRgm6Ea2w])}}
> {{   1161 }}
> {{   1162         for temp_arg in temp_args:}}
> {{/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
> {{     61     def deco(*a, **kw):}}
> {{     62         try:}}
> {{---> 63             return f(*a, **kw)}}
> {{     64         except py4j.protocol.Py4JJavaError as e:}}
> {{     65             s = e.java_exception.toString()}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in 
> get_return_value(answer, gateway_client, target_id, name)}}
> {{    318                 raise Py4JJavaError(}}
> {{    319                     "An error occurred while calling 
> \{0}{1}\{2}.\n".}}
> {{--> 320                     format(target_id, ".", name), value)}}
> {{    321             else:}}
> {{    322                 raise Py4JError(}}
> {{Py4JJavaError: An error occurred while calling o80.get.}}
> {{: java.util.NoSuchElementException: None.get}}
> {{        at scala.None$.get(Option.scala:347)}}
> {{        at scala.None$.get(Option.scala:345)}}
> {{        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{        at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)}}
> {{        at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)}}
> {{        at py4j.Gateway.invoke(Gateway.java:282)}}
> {{        at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)}}
> {{        at py4j.commands.CallCommand.execute(CallCommand.java:79)}}
> {{        at py4j.GatewayConnection.run(GatewayConnection.java:214)}}
> {{        at java.lang.Thread.run(Thread.java:748)}}
>  
> PySpark only prints out the web ui url in `_repr_html`, not `__repr__`, so 
> this only happens in notebooks that render html, not the pyspark shell. 
> [https://github.com/apache/spark/commit/f654b39a63d4f9b118733733c7ed2a1b58649e3d]
>  
> Disabling Spark's UI with `spark.ui.enabled` *is* valuable outside of tests. 
> A couple reasons that come to mind:
> 1) If you run multiple spark applications from one machine, Spark 
> irritatingly starts picking the same port (4040), as the first application, 
> then increments (4041, 4042, etc) until it finds an open port. If you are 
> running 10 spark apps, then the 11th prints out 10 warnings about ports being 
> taken until it finally 

[jira] [Commented] (SPARK-24668) PySpark crashes when getting the webui url if the webui is disabled

[Nikolay Kashtanov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795907#comment-16795907
 ] 

Nikolay Kashtanov commented on SPARK-24668:
---

[~hyukjin.kwon] could you please assign it to me?

> PySpark crashes when getting the webui url if the webui is disabled
> ---
>
> Key: SPARK-24668
> URL: https://issues.apache.org/jira/browse/SPARK-24668
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.3.0, 2.4.0
> Environment: * Spark 2.3.0
>  * Spark-on-YARN
>  * Java 8
>  * Python 3.6.5
>  * Jupyter 4.4.0
>Reporter: Karthik Palaniappan
>Priority: Minor
>
> Repro:
>  
> Evaluate `sc` in a Jupyter notebook:
>  
>  
> {{---}}
> {{Py4JJavaError                             Traceback (most recent call 
> last)}}
> {{/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in 
> __call__(self, obj)}}
> {{    343             method = get_real_method(obj, self.print_method)}}
> {{    344             if method is not None:}}
> {{--> 345                 return method()}}
> {{    346             return None}}
> {{    347         else:}}
> {{/usr/lib/spark/python/pyspark/context.py in _repr_html_(self)}}
> {{    261         }}
> {{    262         """.format(}}
> {{--> 263             sc=self}}
> {{    264         )}}
> {{    265 }}
> {{/usr/lib/spark/python/pyspark/context.py in uiWebUrl(self)}}
> {{    373     def uiWebUrl(self):}}
> {{    374         """Return the URL of the SparkUI instance started by this 
> SparkContext"""}}
> {{--> 375         return 
> self._[jsc.sc|https://www.google.com/url?q=http://jsc.sc=D=AFQjCNHUwO0Cf3OHs1QafBFXzShZ_PU8IQ]().uiWebUrl().get()}}
> {{    376 }}
> {{    377     @property}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in 
> __call__(self, *args)}}
> {{   1158         answer = self.gateway_client.send_command(command)}}
> {{   1159         return_value = get_return_value(}}
> {{-> 1160             answer, self.gateway_client, self.target_id, 
> [self.name|https://www.google.com/url?q=http://self.name=D=AFQjCNEu_LlQOduOrIyV64UgIuRgm6Ea2w])}}
> {{   1161 }}
> {{   1162         for temp_arg in temp_args:}}
> {{/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
> {{     61     def deco(*a, **kw):}}
> {{     62         try:}}
> {{---> 63             return f(*a, **kw)}}
> {{     64         except py4j.protocol.Py4JJavaError as e:}}
> {{     65             s = e.java_exception.toString()}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in 
> get_return_value(answer, gateway_client, target_id, name)}}
> {{    318                 raise Py4JJavaError(}}
> {{    319                     "An error occurred while calling 
> \{0}{1}\{2}.\n".}}
> {{--> 320                     format(target_id, ".", name), value)}}
> {{    321             else:}}
> {{    322                 raise Py4JError(}}
> {{Py4JJavaError: An error occurred while calling o80.get.}}
> {{: java.util.NoSuchElementException: None.get}}
> {{        at scala.None$.get(Option.scala:347)}}
> {{        at scala.None$.get(Option.scala:345)}}
> {{        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{        at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)}}
> {{        at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)}}
> {{        at py4j.Gateway.invoke(Gateway.java:282)}}
> {{        at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)}}
> {{        at py4j.commands.CallCommand.execute(CallCommand.java:79)}}
> {{        at py4j.GatewayConnection.run(GatewayConnection.java:214)}}
> {{        at java.lang.Thread.run(Thread.java:748)}}
>  
> PySpark only prints out the web ui url in `_repr_html`, not `__repr__`, so 
> this only happens in notebooks that render html, not the pyspark shell. 
> [https://github.com/apache/spark/commit/f654b39a63d4f9b118733733c7ed2a1b58649e3d]
>  
> Disabling Spark's UI with `spark.ui.enabled` *is* valuable outside of tests. 
> A couple reasons that come to mind:
> 1) If you run multiple spark applications from one machine, Spark 
> irritatingly starts picking the same port (4040), as the first application, 
> then increments (4041, 4042, etc) until it finds an open port. If you are 
> running 10 spark apps, then the 11th prints out 10 warnings about ports being 
> taken until it finally finds one.
> 2) You can serve the spark web ui 

[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page

[shahid (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795878#comment-16795878
 ] 

shahid edited comment on SPARK-27169 at 3/19/19 9:38 AM:
-

Thank you. Could you please provide full event log if possible? 


was (Author: shahid):
Thank you. Could you please provide full event log if possible? I think, some 
task events are missed in the eventlog

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, 
> job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24668) PySpark crashes when getting the webui url if the webui is disabled

[Nikolay Kashtanov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795890#comment-16795890
 ] 

Nikolay Kashtanov commented on SPARK-24668:
---

Hi [~Karthik Palaniappan] are you worknig on this issue? if not, I'd like to 
take this one.

> PySpark crashes when getting the webui url if the webui is disabled
> ---
>
> Key: SPARK-24668
> URL: https://issues.apache.org/jira/browse/SPARK-24668
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.3.0, 2.4.0
> Environment: * Spark 2.3.0
>  * Spark-on-YARN
>  * Java 8
>  * Python 3.6.5
>  * Jupyter 4.4.0
>Reporter: Karthik Palaniappan
>Priority: Minor
>
> Repro:
>  
> Evaluate `sc` in a Jupyter notebook:
>  
>  
> {{---}}
> {{Py4JJavaError                             Traceback (most recent call 
> last)}}
> {{/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in 
> __call__(self, obj)}}
> {{    343             method = get_real_method(obj, self.print_method)}}
> {{    344             if method is not None:}}
> {{--> 345                 return method()}}
> {{    346             return None}}
> {{    347         else:}}
> {{/usr/lib/spark/python/pyspark/context.py in _repr_html_(self)}}
> {{    261         }}
> {{    262         """.format(}}
> {{--> 263             sc=self}}
> {{    264         )}}
> {{    265 }}
> {{/usr/lib/spark/python/pyspark/context.py in uiWebUrl(self)}}
> {{    373     def uiWebUrl(self):}}
> {{    374         """Return the URL of the SparkUI instance started by this 
> SparkContext"""}}
> {{--> 375         return 
> self._[jsc.sc|https://www.google.com/url?q=http://jsc.sc=D=AFQjCNHUwO0Cf3OHs1QafBFXzShZ_PU8IQ]().uiWebUrl().get()}}
> {{    376 }}
> {{    377     @property}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in 
> __call__(self, *args)}}
> {{   1158         answer = self.gateway_client.send_command(command)}}
> {{   1159         return_value = get_return_value(}}
> {{-> 1160             answer, self.gateway_client, self.target_id, 
> [self.name|https://www.google.com/url?q=http://self.name=D=AFQjCNEu_LlQOduOrIyV64UgIuRgm6Ea2w])}}
> {{   1161 }}
> {{   1162         for temp_arg in temp_args:}}
> {{/usr/lib/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
> {{     61     def deco(*a, **kw):}}
> {{     62         try:}}
> {{---> 63             return f(*a, **kw)}}
> {{     64         except py4j.protocol.Py4JJavaError as e:}}
> {{     65             s = e.java_exception.toString()}}
> {{/usr/lib/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in 
> get_return_value(answer, gateway_client, target_id, name)}}
> {{    318                 raise Py4JJavaError(}}
> {{    319                     "An error occurred while calling 
> \{0}{1}\{2}.\n".}}
> {{--> 320                     format(target_id, ".", name), value)}}
> {{    321             else:}}
> {{    322                 raise Py4JError(}}
> {{Py4JJavaError: An error occurred while calling o80.get.}}
> {{: java.util.NoSuchElementException: None.get}}
> {{        at scala.None$.get(Option.scala:347)}}
> {{        at scala.None$.get(Option.scala:345)}}
> {{        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{        at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)}}
> {{        at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)}}
> {{        at py4j.Gateway.invoke(Gateway.java:282)}}
> {{        at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)}}
> {{        at py4j.commands.CallCommand.execute(CallCommand.java:79)}}
> {{        at py4j.GatewayConnection.run(GatewayConnection.java:214)}}
> {{        at java.lang.Thread.run(Thread.java:748)}}
>  
> PySpark only prints out the web ui url in `_repr_html`, not `__repr__`, so 
> this only happens in notebooks that render html, not the pyspark shell. 
> [https://github.com/apache/spark/commit/f654b39a63d4f9b118733733c7ed2a1b58649e3d]
>  
> Disabling Spark's UI with `spark.ui.enabled` *is* valuable outside of tests. 
> A couple reasons that come to mind:
> 1) If you run multiple spark applications from one machine, Spark 
> irritatingly starts picking the same port (4040), as the first application, 
> then increments (4041, 4042, etc) until it finds an open port. If you are 
> running 10 spark apps, then the 11th prints out 10 warnings about ports being 
> taken until it finally finds 

[jira] [Commented] (SPARK-27052) Using PySpark udf in transform yields NULL values

[Hyukjin Kwon (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795883#comment-16795883
 ] 

Hyukjin Kwon commented on SPARK-27052:
--

It doesn't usually and necessarily assign someone. When you open a PR, it 
automatically assigns. 

> Using PySpark udf in transform yields NULL values
> -
>
> Key: SPARK-27052
> URL: https://issues.apache.org/jira/browse/SPARK-27052
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, SQL
>Affects Versions: 2.4.0
>Reporter: hejsgpuom62c
>Priority: Major
>
> Steps to reproduce
> {code:java}
> from typing import Optional
> from pyspark.sql.functions import expr
> def f(x: Optional[int]) -> Optional[int]:
> return x + 1 if x is not None else None
> spark.udf.register('f', f, "integer")
> df = (spark
> .createDataFrame([(1, [1, 2, 3])], ("id", "xs"))
> .withColumn("xsinc", expr("transform(xs, x -> f(x))")))
> df.show()
> # +---+-+-+
> # | id|   xs|xsinc|
> # +---+-+-+
> # |  1|[1, 2, 3]| [,,]|
> # +---+-+-+
> {code}
>  
> Source https://stackoverflow.com/a/53762650



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

[shahid (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795878#comment-16795878
 ] 

shahid commented on SPARK-27169:


Thank you. Could you please provide full event log if possible? I think, some 
task events are missed in the eventlog

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, 
> job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27194:


Assignee: (was: Apache Spark)

> Job failures when task attempts do not clean up spark-staging parquet files
> ---
>
> Key: SPARK-27194
> URL: https://issues.apache.org/jira/browse/SPARK-27194
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Reza Safi
>Priority: Major
>
> When a container fails for some reason (for example when killed by yarn for 
> exceeding memory limits), the subsequent task attempts for the tasks that 
> were running on that container all fail with a FileAlreadyExistsException. 
> The original task attempt does not seem to successfully call abortTask (or at 
> least its "best effort" delete is unsuccessful) and clean up the parquet file 
> it was writing to, so when later task attempts try to write to the same 
> spark-staging directory using the same file name, the job fails.
> Here is what transpires in the logs:
> The container where task 200.0 is running is killed and the task is lost:
> 19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
> t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 
> GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
>  19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 
> 0.0 (TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 
> exited caused by one of the running tasks) Reason: Container killed by YARN 
> for exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider 
> boosting spark.yarn.executor.memoryOverhead.
> The task is re-attempted on a different executor and fails because the 
> part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
> already exists:
> 19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
> (TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:109)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client 17.161.235.91 already exists
> The job fails when the the configured task attempts (spark.task.maxFailures) 
> have failed with the same error:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 
> in stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 
> 0.0 (TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  ...
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client i.p.a.d already exists
>  
> SPARK-26682 wasn't the root cause here, since there wasn't any stage 
> reattempt.
> This issue seems to happen when 
> spark.sql.sources.partitionOverwriteMode=dynamic. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27194:


Assignee: Apache Spark

> Job failures when task attempts do not clean up spark-staging parquet files
> ---
>
> Key: SPARK-27194
> URL: https://issues.apache.org/jira/browse/SPARK-27194
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Reza Safi
>Assignee: Apache Spark
>Priority: Major
>
> When a container fails for some reason (for example when killed by yarn for 
> exceeding memory limits), the subsequent task attempts for the tasks that 
> were running on that container all fail with a FileAlreadyExistsException. 
> The original task attempt does not seem to successfully call abortTask (or at 
> least its "best effort" delete is unsuccessful) and clean up the parquet file 
> it was writing to, so when later task attempts try to write to the same 
> spark-staging directory using the same file name, the job fails.
> Here is what transpires in the logs:
> The container where task 200.0 is running is killed and the task is lost:
> 19/02/20 09:33:25 ERROR cluster.YarnClusterScheduler: Lost executor y on 
> t.y.z.com: Container killed by YARN for exceeding memory limits. 8.1 GB of 8 
> GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
>  19/02/20 09:33:25 WARN scheduler.TaskSetManager: Lost task 200.0 in stage 
> 0.0 (TID xxx, t.y.z.com, executor 93): ExecutorLostFailure (executor 93 
> exited caused by one of the running tasks) Reason: Container killed by YARN 
> for exceeding memory limits. 8.1 GB of 8 GB physical memory used. Consider 
> boosting spark.yarn.executor.memoryOverhead.
> The task is re-attempted on a different executor and fails because the 
> part-00200-blah-blah.c000.snappy.parquet file from the first task attempt 
> already exists:
> 19/02/20 09:35:01 WARN scheduler.TaskSetManager: Lost task 200.1 in stage 0.0 
> (TID 594, tn.y.z.com, executor 70): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:109)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client 17.161.235.91 already exists
> The job fails when the the configured task attempts (spark.task.maxFailures) 
> have failed with the same error:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 200 
> in stage 0.0 failed 20 times, most recent failure: Lost task 284.19 in stage 
> 0.0 (TID yyy, tm.y.z.com, executor 16): org.apache.spark.SparkException: Task 
> failed while writing rows.
>  at 
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)
>  ...
>  Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> /user/hive/warehouse/tmp_supply_feb1/.spark-staging-blah-blah-blah/dt=2019-02-17/part-00200-blah-blah.c000.snappy.parquet
>  for client i.p.a.d already exists
>  
> SPARK-26682 wasn't the root cause here, since there wasn't any stage 
> reattempt.
> This issue seems to happen when 
> spark.sql.sources.partitionOverwriteMode=dynamic. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27052) Using PySpark udf in transform yields NULL values

[Artem Rybin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795797#comment-16795797
 ] 

Artem Rybin commented on SPARK-27052:
-

Hi [~hejsgpuom62c]!

I reproduced this issue. I would like to investigate this.

Please, assign it to me.

> Using PySpark udf in transform yields NULL values
> -
>
> Key: SPARK-27052
> URL: https://issues.apache.org/jira/browse/SPARK-27052
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, SQL
>Affects Versions: 2.4.0
>Reporter: hejsgpuom62c
>Priority: Major
>
> Steps to reproduce
> {code:java}
> from typing import Optional
> from pyspark.sql.functions import expr
> def f(x: Optional[int]) -> Optional[int]:
> return x + 1 if x is not None else None
> spark.udf.register('f', f, "integer")
> df = (spark
> .createDataFrame([(1, [1, 2, 3])], ("id", "xs"))
> .withColumn("xsinc", expr("transform(xs, x -> f(x))")))
> df.show()
> # +---+-+-+
> # | id|   xs|xsinc|
> # +---+-+-+
> # |  1|[1, 2, 3]| [,,]|
> # +---+-+-+
> {code}
>  
> Source https://stackoverflow.com/a/53762650



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27199) Replace TimeZone by ZoneId in TimestampFormatter API

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27199:


Assignee: Apache Spark

> Replace TimeZone by ZoneId in TimestampFormatter API
> 
>
> Key: SPARK-27199
> URL: https://issues.apache.org/jira/browse/SPARK-27199
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Maxim Gekk
>Assignee: Apache Spark
>Priority: Minor
>
> Internally, TimestampFormatter implementations use ZoneId but not TimeZone 
> which comes via API. Conversion from TimeZone to ZoneId is not for free. 
> Actually, TimeZone is converted to String, and the String and parsed to 
> ZoneId. The conversion to String can be eliminated if TimestampFormatter 
> would accept ZoneId. And also, TimeZone is converted from String in some 
> cases (JSON options). So, in bad case String -> TimeZone -> String -> ZoneId 
> -> ZoneOffset. The ticket aims to use ZoneId in TimestampFormatter API. We 
> could require ZoneOffset but it is not convenient in most cases because 
> conversion ZoneId to ZoneOffset requires Instant. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27199) Replace TimeZone by ZoneId in TimestampFormatter API

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-27199:


Assignee: (was: Apache Spark)

> Replace TimeZone by ZoneId in TimestampFormatter API
> 
>
> Key: SPARK-27199
> URL: https://issues.apache.org/jira/browse/SPARK-27199
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> Internally, TimestampFormatter implementations use ZoneId but not TimeZone 
> which comes via API. Conversion from TimeZone to ZoneId is not for free. 
> Actually, TimeZone is converted to String, and the String and parsed to 
> ZoneId. The conversion to String can be eliminated if TimestampFormatter 
> would accept ZoneId. And also, TimeZone is converted from String in some 
> cases (JSON options). So, in bad case String -> TimeZone -> String -> ZoneId 
> -> ZoneOffset. The ticket aims to use ZoneId in TimestampFormatter API. We 
> could require ZoneOffset but it is not convenient in most cases because 
> conversion ZoneId to ZoneOffset requires Instant. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795774#comment-16795774
 ] 

acupple edited comment on SPARK-27169 at 3/19/19 7:22 AM:
--

I have enabled event log, on the spark UI, the job has not completed, but 
according to event log, the job has completed.

!image-2019-03-19-15-17-25-522.png|width=872,height=48!

the attachment is event log

[^job_1924.log]

And there is one unknown stage

!image-2019-03-19-15-21-03-766.png|width=610,height=70!

 


was (Author: acupple):
I have enabled event log, on the spark UI, the job has not completed, but 
according to event log, the job has completed.

!image-2019-03-19-15-17-25-522.png|width=872,height=48!

the attachment is event log

[^job_1924.log]

And there is one unknown stage

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, 
> job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795774#comment-16795774
 ] 

acupple commented on SPARK-27169:
-

I have enabled event log, on the spark UI, the job has not completed, but 
according to event log, the job has completed.

!image-2019-03-19-15-17-25-522.png|width=872,height=48!

the attachment is event log

[^job_1924.log]

And there is one unknown stage

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

acupple updated SPARK-27169:

Attachment: image-2019-03-19-15-17-25-522.png

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27169) number of active tasks is negative on executors page

[acupple (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

acupple updated SPARK-27169:

Attachment: job_1924.log

> number of active tasks is negative on executors page
> 
>
> Key: SPARK-27169
> URL: https://issues.apache.org/jira/browse/SPARK-27169
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.3.2
>Reporter: acupple
>Priority: Minor
> Attachments: QQ20190315-102215.png, QQ20190315-102235.png, 
> image-2019-03-19-15-17-25-522.png, job_1924.log
>
>
> I use spark to process some data in HDFS and HBASE, I use one thread consume 
> message from a queue, and then submit to a thread pool（16 fix size）for spark 
> processor.
> But when run for some time, the active jobs will be thousands, and number of 
> active tasks are negative.
> Actually, these jobs are already done when I check driver logs。
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27199) Replace TimeZone by ZoneId in TimestampFormatter API

[Maxim Gekk (JIRA)]

Maxim Gekk created SPARK-27199:
--

 Summary: Replace TimeZone by ZoneId in TimestampFormatter API
 Key: SPARK-27199
 URL: https://issues.apache.org/jira/browse/SPARK-27199
 Project: Spark
  Issue Type: Improvement
  Components: SQL
Affects Versions: 2.4.0
Reporter: Maxim Gekk


Internally, TimestampFormatter implementations use ZoneId but not TimeZone 
which comes via API. Conversion from TimeZone to ZoneId is not for free. 
Actually, TimeZone is converted to String, and the String and parsed to ZoneId. 
The conversion to String can be eliminated if TimestampFormatter would accept 
ZoneId. And also, TimeZone is converted from String in some cases (JSON 
options). So, in bad case String -> TimeZone -> String -> ZoneId -> ZoneOffset. 
The ticket aims to use ZoneId in TimestampFormatter API. We could require 
ZoneOffset but it is not convenient in most cases because conversion ZoneId to 
ZoneOffset requires Instant. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-27193) CodeFormatter should format multi comment lines correctly

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan resolved SPARK-27193.
-
   Resolution: Fixed
Fix Version/s: 3.0.0

Issue resolved by pull request 24133
[https://github.com/apache/spark/pull/24133]

> CodeFormatter should format multi comment lines correctly
> -
>
> Key: SPARK-27193
> URL: https://issues.apache.org/jira/browse/SPARK-27193
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: wuyi
>Assignee: wuyi
>Priority: Trivial
> Fix For: 3.0.0
>
>
> when enable `spark.sql.codegen.comments`,  there will be multiple comment 
> lines. However, CodeFormatter can not handle multi comment lines currently:
>  
> Generated code:
> /* 001 */ public Object generate(Object[] references) {
> /* 002 */   return new GeneratedIteratorForCodegenStage1(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ /**
> \* Codegend pipeline for stage (id=1)
> \* *(1) Project [(id#0L + 1) AS (id + 1)#3L]
> \* +- *(1) Filter (id#0L = 1)
> \*+- *(1) Range (0, 10, step=1, splits=4)
> \*/
> /* 006 */ // codegenStageId=1
> /* 007 */ final class GeneratedIteratorForCodegenStage1 extends 
> org.apache.spark.sql.execution.BufferedRowIterator {



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-27193) CodeFormatter should format multi comment lines correctly

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-27193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan reassigned SPARK-27193:
---

Assignee: wuyi

> CodeFormatter should format multi comment lines correctly
> -
>
> Key: SPARK-27193
> URL: https://issues.apache.org/jira/browse/SPARK-27193
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: wuyi
>Assignee: wuyi
>Priority: Trivial
>
> when enable `spark.sql.codegen.comments`,  there will be multiple comment 
> lines. However, CodeFormatter can not handle multi comment lines currently:
>  
> Generated code:
> /* 001 */ public Object generate(Object[] references) {
> /* 002 */   return new GeneratedIteratorForCodegenStage1(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ /**
> \* Codegend pipeline for stage (id=1)
> \* *(1) Project [(id#0L + 1) AS (id + 1)#3L]
> \* +- *(1) Filter (id#0L = 1)
> \*+- *(1) Range (0, 10, step=1, splits=4)
> \*/
> /* 006 */ // codegenStageId=1
> /* 007 */ final class GeneratedIteratorForCodegenStage1 extends 
> org.apache.spark.sql.execution.BufferedRowIterator {



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org