Re: Spark master build hangs using parallel build option in maven

Hi, Saurabh. It seems that you are hitting https://issues.apache.org/jira/browse/SPARK-26095 . And, we disabled the parallel build via https://github.com/apache/spark/pull/23061 at 3.0.0. According to the stack trace in JIRA and PR description, `maven-shade-plugin` seems to be the root cause.

Re: Spark master build hangs using parallel build option in maven

Hi Sean, Thanks for checking this. I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark " You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3"

Subscribe to spark-dev

Please add me to spark-dev mailing list.

Re: Spark master build hangs using parallel build option in maven

I don't believe you can use a parallel build indeed. Some things collide with each other. Some of the suites are run in parallel inside the build though already. On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla wrote: > > Hi All, > > Spark master build hangs using parallel build option in maven.

unsubscribe

- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Spark master build hangs using parallel build option in maven

Hi All, Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (*-Phadoop-provided -Dhadoop.version=2.8.5) *option. Same command works fine to build spark-2.4.3

Re: [Discuss] Metrics Support for DS V2

We've implemented these metrics in the RDD (for input metrics) and in the v2 DataWritingSparkTask. That approach gives you the same metrics in the stage views that you get with v1 sources, regardless of the v2 implementation. I'm not sure why they weren't included from the start. It looks like

Re: [Discuss] Metrics Support for DS V2

I think there are a few details we need to discuss. how frequently a source should update its metrics? For example, if file source needs to report size metrics per row, it'll be super slow. what metrics a source should report? data size? numFiles? read time? shall we show metrics in SQL web UI

unsubscribe

Re: How to implement a "saveAsBinaryFile" function?

Hi Fokko, Maxim, Long: Thanks! This reading has been occurred in a custom datasource as below: override def createRelation(…) { … blocks.map(block => (block.bytes)).saveAsTextFile(parameters("path”)) ... } I am a new Sparker, will try the those methods you guys provides. Best! Bing. On Jan