Re: [VOTE] SPARK 2.4.0 (RC2)

+1, with comments: There are 5 critical issues for 2.4, and no blockers: SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4 SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella SPARK-25326 ML, Graph 2.4 QA:

[DISCUSS] Syntax for table DDL

Hi everyone, I’m currently working on new table DDL statements for v2 tables. For context, the new logical plans for DataSourceV2 require a catalog interface so that Spark can create tables for operations like CTAS. The proposed TableCatalog API also includes an API for altering those tables so

On Scala 2.12.7

I'm forking the discussion about Scala 2.12.7 from the 2.4.0 RC vote thread. 2.12.7 was released yesterday, and, is even labeled as fixing Spark 2.4 compatibility! https://www.scala-lang.org/news/2.12.7 We should look into it, yes. Darcy identified, and they fixed, this issue:

Re: [DISCUSS] SPIP: Native support of session window

Btw, just wrote up detailed design doc on existing patch: https://docs.google.com/document/d/1tUO29BDXb9127RiivUS7Hv324dC0YHuokYvyQRpurDY/edit?usp=sharing This doc is a wall of text, since I guess we already imagine how session window works (and I showed a simple example in SPIP doc), so try to

Re: [VOTE] SPARK 2.4.0 (RC2)

Go ahead and file a JIRA to update to 2.12.7 with these details. We'll assess whether it is a blocker. On Fri, Sep 28, 2018 at 12:09 PM Darcy Shen wrote: > > I agree it is a non-important Spark bug. I mean the Option and String > comparison. The bug is easy to fix and obvious to confirm. If the

Re: [VOTE] SPARK 2.4.0 (RC2)

I agree it is a non-important Spark bug. I mean the Option and String comparison. The bug is easy to fix and obvious to confirm. If the desc of PR may not be accurate，feel free to edit the title or content. I am on a vocation from 9.29   ：）But the scala bug of

Re: [VOTE] SPARK 2.4.0 (RC2)

Hi, Wenchen. The current issue link seems to be out of order for me. The list of bug fixes going into 2.4.0 can be found at the following URL: https://issues.apache.org/jira/projects/SPARK/versions/2.4.0 Could you send out with the following issue link for next RCs?

Re: [VOTE] SPARK 2.4.0 (RC2)

I don't even know how or if this manifests as a bug. The code is indeed incorrect and the 2.12 compiler flags it. We fixed a number of these in SPARK-25398. While I want to get this into 2.4 if we have another RC, I don't see evidence this is a blocker. It is not specific to Scala 2.12. Using

Re: [VOTE] SPARK 2.4.0 (RC2)

I don't think this bug is so serious to fail an RC, it's only about metrics IIUC, and it's not a regression in 2.4. I agree we should backport this fix to 2.3 and 2.4, and we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to change it? BTW end users can

I want read text cluomn from kafka topic pyspark

I write code to read data from twitter and send data to kafka topic and I write anther code to read data from kafka topic I want to return just text column from data -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: [DISCUSS] SPIP: Native support of session window

Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned! Btw, I'm trying out your patch to see whether it passes the tests I've added, and

Re: [DISCUSS] SPIP: Native support of session window

Hi Jungtaek: We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product. As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see