Re: [Question] Can't start Spark Connect

[Haejoon Lee]

Additionally, try deleting the `.idea` in the spark home directory and
restarting IntelliJ if it does not work properly after re-building during
development.
The .idea stores IntelliJ's project configuration and settings, and is
automatically generated when IntelliJ is launched.

>

Re: [Question] Can't start Spark Connect

[yangjie01]

Yeah, after executing `./build/mvn -DskipTests clean package `  on the command 
line, you may need to manually reload maven projects in intellij, otherwise 
intellij will not immediately respond to the command line behavior.

Yang Jie

发件人: Lucifer Tyrant 
日期: 2023年3月9日 星期四 00:11
收件人: Jia Fan 
抄送: Herman van Hovell , Hyukjin Kwon 
, "dev@spark.apache.org" 
主题: Re: [Question] Can't start Spark Connect

It worked. Thank you for your guidance.

Jia Fan mailto:fan...@apache.org>> 于2023年3月8日周三 22:38写道：
Hi Herman,
I just use ./build/mvn -DskipTests clean package, also had try to use 
./build/mvn -DskipTests clean install.

Herman van Hovell  于2023年3月8日周三 21:17写道：
Hi Jia,

How are you building connect?

Kind regards,
Herman

On Wed, Mar 8, 2023 at 8:48 AM Jia Fan 
mailto:fan...@apache.org>> wrote:
Thanks for reply,
I had done clean build with maven few times. But always report
/Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
java: symbol not found
   Symbol: class UnusedPrivateParameter
   Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
   I think maybe it's protobuf version conflict?
https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png">


Hyukjin Kwon mailto:gurwls...@gmail.com>> 于2023年3月8日周三 
19:09写道：
Just doing a clean build with Maven, and running a test case like 
`SparkConnectServiceSuite` in IntelliJ should work.

On Wed, 8 Mar 2023 at 15:02, Jia Fan 
mailto:fan...@apache.org>> wrote:
Hi developers,
   I want to contribute some code for Spark Connect. Any doc for starters? I 
want to debug SimpleSparkConnectService but I can't start it with IDEA. I would 
appreciate any help.

Thanks





Jia Fan

Re: [Question] Can't start Spark Connect

[Lucifer Tyrant]

It worked. Thank you for your guidance.

Jia Fan  于2023年3月8日周三 22:38写道：

> Hi Herman,
> I just use ./build/mvn -DskipTests clean package, also had try to
> use ./build/mvn -DskipTests clean install.
>
> Herman van Hovell  于2023年3月8日周三 21:17写道：
>
>> Hi Jia,
>>
>> How are you building connect?
>>
>> Kind regards,
>> Herman
>>
>> On Wed, Mar 8, 2023 at 8:48 AM Jia Fan  wrote:
>>
>>> Thanks for reply,
>>> I had done clean build with maven few times. But always report
>>>
>>> /Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
>>> java: symbol not found
>>>Symbol: class UnusedPrivateParameter
>>>Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
>>>I think maybe it's protobuf version conflict?
>>> https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png
>>> ">
>>>
>>>
>>> Hyukjin Kwon  于2023年3月8日周三 19:09写道：
>>>
 Just doing a clean build with Maven, and running a test case like
 `SparkConnectServiceSuite` in IntelliJ should work.

 On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:

> Hi developers,
>I want to contribute some code for Spark Connect. Any doc for
> starters? I want to debug SimpleSparkConnectService but I can't start it
> with IDEA. I would appreciate any help.
>
> Thanks
>
> 
>
>
> Jia Fan
>

Re: Undelivered Mail Returned to Sender

[Jia Fan]

Hi guys,
  After I reload and rebuild project. Finally I run test case successfully.
Thanks for you help, Herman and Hyukjin.

Mail Delivery System  于2023年3月8日周三
22:38写道：

> This is the mail system at host mxout1-he-de.apache.org.
>
> I'm sorry to have to inform you that your message could not
> be delivered to one or more recipients. It's attached below.
>
> For further assistance, please send mail to postmaster.
>
> If you do so, please include this problem report. You can
> delete your own text from the attached returned message.
>
>The mail system
>
> : Host or domain name not found. Name
> service
> error for name=databricks.com.invalid type=: Host not found
>
>
>
> -- Forwarded message --
> From: Jia Fan 
> To: Herman van Hovell 
> Cc: Hyukjin Kwon , dev@spark.apache.org
> Bcc:
> Date: Wed, 8 Mar 2023 22:37:33 +0800
> Subject: Re: [Question] Can't start Spark Connect
> Hi Herman,
> I just use ./build/mvn -DskipTests clean package, also had try to
> use ./build/mvn -DskipTests clean install.
>
> Herman van Hovell  于2023年3月8日周三 21:17写道：
>
>> Hi Jia,
>>
>> How are you building connect?
>>
>> Kind regards,
>> Herman
>>
>> On Wed, Mar 8, 2023 at 8:48 AM Jia Fan  wrote:
>>
>>> Thanks for reply,
>>> I had done clean build with maven few times. But always report
>>>
>>> /Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
>>> java: symbol not found
>>>Symbol: class UnusedPrivateParameter
>>>Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
>>>I think maybe it's protobuf version conflict?
>>> https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png
>>> ">
>>>
>>>
>>> Hyukjin Kwon  于2023年3月8日周三 19:09写道：
>>>
 Just doing a clean build with Maven, and running a test case like
 `SparkConnectServiceSuite` in IntelliJ should work.

 On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:

> Hi developers,
>I want to contribute some code for Spark Connect. Any doc for
> starters? I want to debug SimpleSparkConnectService but I can't start it
> with IDEA. I would appreciate any help.
>
> Thanks
>
> 
>
>
> Jia Fan
>

Re: [Question] Can't start Spark Connect

[Jia Fan]

Hi Herman,
I just use ./build/mvn -DskipTests clean package, also had try to
use ./build/mvn -DskipTests clean install.

Herman van Hovell  于2023年3月8日周三 21:17写道：

> Hi Jia,
>
> How are you building connect?
>
> Kind regards,
> Herman
>
> On Wed, Mar 8, 2023 at 8:48 AM Jia Fan  wrote:
>
>> Thanks for reply,
>> I had done clean build with maven few times. But always report
>>
>> /Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
>> java: symbol not found
>>Symbol: class UnusedPrivateParameter
>>Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
>>I think maybe it's protobuf version conflict?
>> https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png
>> ">
>>
>>
>> Hyukjin Kwon  于2023年3月8日周三 19:09写道：
>>
>>> Just doing a clean build with Maven, and running a test case like
>>> `SparkConnectServiceSuite` in IntelliJ should work.
>>>
>>> On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:
>>>
 Hi developers,
I want to contribute some code for Spark Connect. Any doc for
 starters? I want to debug SimpleSparkConnectService but I can't start it
 with IDEA. I would appreciate any help.

 Thanks

 


 Jia Fan

>>>

Re: [Question] Can't start Spark Connect

[Herman van Hovell]

Hi Jia,

How are you building connect?

Kind regards,
Herman

On Wed, Mar 8, 2023 at 8:48 AM Jia Fan  wrote:

> Thanks for reply,
> I had done clean build with maven few times. But always report
>
> /Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
> java: symbol not found
>Symbol: class UnusedPrivateParameter
>Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
>I think maybe it's protobuf version conflict?
> https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png
> ">
>
>
> Hyukjin Kwon  于2023年3月8日周三 19:09写道：
>
>> Just doing a clean build with Maven, and running a test case like
>> `SparkConnectServiceSuite` in IntelliJ should work.
>>
>> On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:
>>
>>> Hi developers,
>>>I want to contribute some code for Spark Connect. Any doc for
>>> starters? I want to debug SimpleSparkConnectService but I can't start it
>>> with IDEA. I would appreciate any help.
>>>
>>> Thanks
>>>
>>> 
>>>
>>>
>>> Jia Fan
>>>
>>

Re: Re: [DISCUSS][SPIP] Subexpression elimination supporting more physical operators

[Adam Binford]

I'm +1 for anything that adds support for subexpression elimination to
conditional expressions, I've been trying to get that support in for a long
time https://github.com/apache/spark/pull/32987. This is a major
performance limitation in Spark in my opinion.

On Tue, Mar 7, 2023 at 4:24 PM Kun Wan  wrote:

> Hi, Mich
> Thanks for your reminder.
> It seems "Support for subexpression elimination with more physical
> operators" should not be a SPIP, it's just a design doc.
> I have changed the title of the document.
> Hope someone who are interested will join in and complete it together.
>
> At 2023-03-08 01:50:17, "Mich Talebzadeh" 
> wrote:
>
>
> Hi Kun,
>
>
> Have you checked the doc procedure for SPIP here in case
>
>
> Spark Project Improvement Proposals (SPIP) | Apache Spark
> 
>
>
> HT
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Tue, 7 Mar 2023 at 13:32, 万昆  wrote:
>
>> Hi everyone,
>>
>>
>> I would like to start a discussion on "Subexpression elimination
>> supporting more physical operators"
>>
>>
>> Now spark only support subexpression elimination in ProjectExec and
>> AggregateExec. We can support improve subexpression elimination framework
>> to support more physical operators.
>>
>>
>> SPIP Jira: https://issues.apache.org/jira/browse/SPARK-42551
>>
>> SPIP Doc:
>> https://docs.google.com/document/d/165cv7hRvkFvuUHlnbapWvamxcxcn9FJj/edit?usp=sharing=107277827304520252190=true=true
>>
>>
>> Thank you
>>
>>

-- 
Adam Binford

Re: [Question] Can't start Spark Connect

[Jia Fan]

Thanks for reply,
I had done clean build with maven few times. But always report
/Users/xxx/Code/spark/core/target/generated-sources/org/apache/spark/status/protobuf/StoreTypes.java:658:9
java: symbol not found
   Symbol: class UnusedPrivateParameter
   Location: class org.apache.spark.status.protobuf.StoreTypes.JobData
   I think maybe it's protobuf version conflict?
https://user-images.githubusercontent.com/32387433/223716946-85761a34-f86c-4ba1-9557-a59d0d5b9958.png
">


Hyukjin Kwon  于2023年3月8日周三 19:09写道：

> Just doing a clean build with Maven, and running a test case like
> `SparkConnectServiceSuite` in IntelliJ should work.
>
> On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:
>
>> Hi developers,
>>I want to contribute some code for Spark Connect. Any doc for
>> starters? I want to debug SimpleSparkConnectService but I can't start it
>> with IDEA. I would appreciate any help.
>>
>> Thanks
>>
>> 
>>
>>
>> Jia Fan
>>
>

Re: [Question] Can't start Spark Connect

[Hyukjin Kwon]

Just doing a clean build with Maven, and running a test case like
`SparkConnectServiceSuite` in IntelliJ should work.

On Wed, 8 Mar 2023 at 15:02, Jia Fan  wrote:

> Hi developers,
>I want to contribute some code for Spark Connect. Any doc for starters?
> I want to debug SimpleSparkConnectService but I can't start it with IDEA. I
> would appreciate any help.
>
> Thanks
>
> 
>
>
> Jia Fan
>