Re: ClassNotFoundException while running unit test with local cluster mode in Intellij IDEA

Hi, cloud0fan. Yeah, tests run well in SBT. Maybe, I should try your way. Thanks! -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: ClassNotFoundException while running unit test with local cluster mode in Intellij IDEA

Hi, cloud0fan, I tried it and that's really good and cool! Thanks again! -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

ClassNotFoundException while running unit test with local cluster mode in Intellij IDEA

when I try to run unit test with *local cluster* in *Intellij IDEA*. I'd appreciate a lot if any guys could give me a hint. THANKS. Best wishes. wuyi -- Sent from: http://apache-spark-developers-list.1001551.n3.nabb

Cannot run program ".../jre/bin/javac": error=2, No such file or directory

incremental compilation [INFO] Toolchain in scala-maven-plugin: */Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre* [warn] Pruning sources from previous analysis, due to incompatible CompileSetup. [info] Compiling 2 Scala sources and 8 Java sources to /Users/wuyi/workspace/spa

Re: Cannot run program ".../jre/bin/javac": error=2, No such file or directory

Hi, Owen, thank for your suggestion. I recheck my env and do not find any wrong with JAVA_HOME. But I'm agree with you there must be something wrong with the system env. Currently, I created a link file (named javac) under $JAVA_HOME/jre/bin to link to $JAVA_HOME/bin/javac to work around

[DISCUSS] Introduce WorkerOffer reservation mechanism for barrier scheduling

Currently, Barrier TaskSet has a hard requirement that tasks can only be launched in a single resourceOffers round with enough slots(or say, sufficient resources), but can not be guaranteed even if with enough slots due to task locality delay scheduling(also see discussion

Re: [DISCUSS] Null-handling of primitive-type of untyped Scala UDF in Scala 2.12

Hi Takeshi, thanks for your reply. Before the broken, we only do the null check for primitive types and leave null value of non-primitive type to UDF itself in case it will be handled specifically, e.g., a UDF may return something else for null String. -- Sent from:

Re: BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?

Hi Jacek, code in link [2] is the out of date. The commit https://github.com/apache/spark/commit/32ec528e63cb768f85644282978040157c3c2fb7 has already removed unreachable branch. Best, Yi Wu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: spark lacks fault tolerance with dynamic partition overwrite

Hi, Koert, The community is back to this issue to recently and there's already a fix https://github.com/apache/spark/pull/26339 for it. You can track and review it there. Best, Yi Wu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

[DISCUSS] Null-handling of primitive-type of untyped Scala UDF in Scala 2.12

Hi all, I'd like to raise a discussion here about null-handling of primitive-type of untyped Scala UDF [ udf(f: AnyRef, dataType: DataType) ]. After we switch to Scala 2.12 in 3.0, the untyped Scala UDF is broken because now we can't use reflection to get the parameter types of the Scala lambda.

Re: Getting the ball started on a 2.4.6 release

We have a conclusion now and we decide to include SPARK-31509 in the PR of SPARK-31485. So there actually should be only one candidate(But to be honest, it still depends on committers). Best, Yi Wu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could cause application hang. And, probably, also https://issues.apache.org/jira/browse/SPARK-31509, to make better guidance of barrier execution for user. But we do not have conclusion yet. Best, Yi Wu -- Sent from:

Re: [DISCUSS] Null-handling of primitive-type of untyped Scala UDF in Scala 2.12

Thanks Sean and Takeshi. Option 1 seems really impossible. And I'm going to take Option 2 as an alternative choice. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail:

Re: Apache Spark 3.1 Feature Expectation (Dec. 2020)

This could be a sub-task of https://issues.apache.org/jira/browse/SPARK-25299 (Use remote storage for persisting shuffle data)? It's good if we could put the whole SPARK-25299 in Spark 3.1. Holden Karau wrote > Should we also consider the

Re: [VOTE] Decommissioning SPIP

+1 for having this feature in Spark -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[OSS DIGEST] The major changes of Apache Spark from May 20 to June 2

Hi all, This is the bi-weekly Apache Spark digest from the Databricks OSS team. For each API/configuration/behavior change, an [API] tag is added in the title. Hi all, This is the bi-weekly Apache Spark digest from the Databricks OSS team. For each API/configuration/behavior change, an [API]

Re: [DISCUSS] Apache Spark 3.0.1 Release

Ok, after having another look, I think it only affects local cluster deploy mode, which is for testing only. wuyi wrote > Please also includes https://issues.apache.org/jira/browse/SPARK-32120 in > Spark 3.0.1. It's a regression compares to Spark 3.0.0-preview2. > > Tha

Re: UnknownSource NullPointerException in CodeGen. with Custom Strategy

Hi Nasrulla, Could you give a complete demo to reproduce the issue? -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS][SPIP] Graceful Decommissioning

Thank you for your effort, Holden. I left a few comments in SPIP. I asked for some details, though I know some contents have been include in the design doc. I'm not very clear about difference between the design doc and SPIP. But from what I saw at the SPIP template questions, I think some

Re: [DISCUSS][SPIP] Graceful Decommissioning

2020 at 1:38 AM wuyi > yi.wu@ > wrote: > >> Thank you for your effort, Holden. >> >> I left a few comments in SPIP. I asked for some details, though I know >> some >> contents have been include in the design doc. I'm not very clear about >> dif

Re: [ANNOUNCE] Apache Spark 3.0.0

Congrats!! -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] Apache Spark 3.0.1 Release

Please also includes https://issues.apache.org/jira/browse/SPARK-32120 in Spark 3.0.1. It's a regression compares to Spark 3.0.0-preview2. Thanks, Yi Wu Yuanjian Li wrote > Hi dev-list, > > I’m writing this to raise the discussion about Spark 3.0.1 feasibility > since 4 blocker issues were