Re: Unable to resolve dependency of sbt-mima-plugin since yesterday

[Xiao Li]

Thank you, Dongjoon!

Xiao

On Tue, Oct 22, 2019 at 5:08 PM Dongjoon Hyun 
wrote:

> Hi, All.
>
> This is fixed in master/branch-2.4.
>
> Bests,
> Dongjoon.
>
> On Tue, Oct 22, 2019 at 12:19 Sean Owen  wrote:
>
>> Weird. Let's discuss at https://issues.apache.org/jira/browse/SPARK-29560
>>
>> On Tue, Oct 22, 2019 at 2:06 PM Xingbo Jiang 
>> wrote:
>> >
>> > Hi,
>> >
>> > Do you have any idea why the `./dev/lint-scala` check are failure with
>> the following message since yesterday ？
>> >
>> >> WARNING: An illegal reflective access operation has occurred
>> >> 9WARNING: Illegal reflective access by
>> org.apache.ivy.util.url.IvyAuthenticator
>> (file:/home/runner/work/spark/spark/build/sbt-launch-0.13.18.jar) to field
>> java.net.Authenticator.theAuthenticator
>> >> 10WARNING: Please consider reporting this to the maintainers of
>> org.apache.ivy.util.url.IvyAuthenticator
>> >> 11WARNING: Use --illegal-access=warn to enable warnings of further
>> illegal reflective access operations
>> >> 12WARNING: All illegal access operations will be denied in a future
>> release
>> >> 13Scalastyle checks failed at following occurrences:
>> >> 14[error] (*:update) sbt.ResolveException: unresolved dependency:
>> com.typesafe#sbt-mima-plugin;0.3.0: not found
>> >> 15##[error]Process completed with exit code 1.
>> >
>> >
>> > I'm not able to reproduce the failure on my local environment, but
>> seems all the open PRs are failing on this check.
>> >
>> > Thanks,
>> >
>> > Xingbo
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

-- 
[image: Databricks Summit - Watch the talks]

Re: Unable to resolve dependency of sbt-mima-plugin since yesterday

[Dongjoon Hyun]

Hi, All.

This is fixed in master/branch-2.4.

Bests,
Dongjoon.

On Tue, Oct 22, 2019 at 12:19 Sean Owen  wrote:

> Weird. Let's discuss at https://issues.apache.org/jira/browse/SPARK-29560
>
> On Tue, Oct 22, 2019 at 2:06 PM Xingbo Jiang 
> wrote:
> >
> > Hi,
> >
> > Do you have any idea why the `./dev/lint-scala` check are failure with
> the following message since yesterday ？
> >
> >> WARNING: An illegal reflective access operation has occurred
> >> 9WARNING: Illegal reflective access by
> org.apache.ivy.util.url.IvyAuthenticator
> (file:/home/runner/work/spark/spark/build/sbt-launch-0.13.18.jar) to field
> java.net.Authenticator.theAuthenticator
> >> 10WARNING: Please consider reporting this to the maintainers of
> org.apache.ivy.util.url.IvyAuthenticator
> >> 11WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> >> 12WARNING: All illegal access operations will be denied in a future
> release
> >> 13Scalastyle checks failed at following occurrences:
> >> 14[error] (*:update) sbt.ResolveException: unresolved dependency:
> com.typesafe#sbt-mima-plugin;0.3.0: not found
> >> 15##[error]Process completed with exit code 1.
> >
> >
> > I'm not able to reproduce the failure on my local environment, but seems
> all the open PRs are failing on this check.
> >
> > Thanks,
> >
> > Xingbo
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Question about in subquery

[zhangliyun]

Hi all:
   i  used in subquery like following in spark 2.3.1
{code}
set spark.sql.autoBroadcastJoinThreshold=-1;


explain select * from testdata where key1 not in (select key1 from testdata as 
b);




= Physical Plan ==


BroadcastNestedLoopJoin BuildRight, LeftAnti, ((key1#60 = key1#62) || 
isnull((key1#60 = key1#62)))


:- HiveTableScan [key1#60, value1#61], HiveTableRelation `default`.`testdata`, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [key1#60, value1#61]


+- BroadcastExchange IdentityBroadcastMode


   +- HiveTableScan [key1#62], HiveTableRelation `default`.`testdata`, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [key1#62, value1#63]
{code}


It seems that even i disable the broadcast join ( set 
spark.sql.autoBroadcastJoinThreshold=-1). it still a  broadcast join not sort 
merged join. Is there any way to let spark sql to use sort merge join by 
setting parameters besides directly modifying the sql like : explain select 
testdata.* from testdata left join testdata2 where testdata.key1=testdata2.key1 
and testdata2.key1 is null;






Appreciate to have some suggestions


Best Regards


Kelly Zhang

[DISCUSS] commit to ExpressionEncoder?

[Erik Erlandson]

Currently the design of Encoder implies the possibility that encoders might
be customized, or at least that there are other internal alternatives to
ExpressionEncoder. However there are both implicit and explicit
restrictions in spark-sql, such that ExpressionEncoder is the only
functional option, including but not limited to:

  def encoderFor[A : Encoder]: ExpressionEncoder[A] =
implicitly[Encoder[A]] match {
case e: ExpressionEncoder[A] =>
  e.assertUnresolved()
  e
case _ => sys.error(s"Only expression encoders are supported today")
  }

My impression, based on recent work around the SQL code, is that:
1) ExpressionEncoder does support the space of plausible data types and
objects, including custom ones.
2) Supporting other subclasses of Encoder in actuality would be nontrivial

Should spark SQL more explicitly commit to an "ExpressionEncoder only"
position?

Re: Unable to resolve dependency of sbt-mima-plugin since yesterday

[Sean Owen]

Weird. Let's discuss at https://issues.apache.org/jira/browse/SPARK-29560

On Tue, Oct 22, 2019 at 2:06 PM Xingbo Jiang  wrote:
>
> Hi,
>
> Do you have any idea why the `./dev/lint-scala` check are failure with the 
> following message since yesterday ？
>
>> WARNING: An illegal reflective access operation has occurred
>> 9WARNING: Illegal reflective access by 
>> org.apache.ivy.util.url.IvyAuthenticator 
>> (file:/home/runner/work/spark/spark/build/sbt-launch-0.13.18.jar) to field 
>> java.net.Authenticator.theAuthenticator
>> 10WARNING: Please consider reporting this to the maintainers of 
>> org.apache.ivy.util.url.IvyAuthenticator
>> 11WARNING: Use --illegal-access=warn to enable warnings of further illegal 
>> reflective access operations
>> 12WARNING: All illegal access operations will be denied in a future release
>> 13Scalastyle checks failed at following occurrences:
>> 14[error] (*:update) sbt.ResolveException: unresolved dependency: 
>> com.typesafe#sbt-mima-plugin;0.3.0: not found
>> 15##[error]Process completed with exit code 1.
>
>
> I'm not able to reproduce the failure on my local environment, but seems all 
> the open PRs are failing on this check.
>
> Thanks,
>
> Xingbo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unable to resolve dependency of sbt-mima-plugin since yesterday

[Xingbo Jiang]

Hi,

Do you have any idea why the `./dev/lint-scala` check are failure with the
following message since yesterday ？

WARNING: An illegal reflective access operation has occurred
> 9
> WARNING:
> Illegal reflective access by org.apache.ivy.util.url.IvyAuthenticator
> (file:/home/runner/work/spark/spark/build/sbt-launch-0.13.18.jar) to field
> java.net.Authenticator.theAuthenticator
> 10
> WARNING:
> Please consider reporting this to the maintainers of
> org.apache.ivy.util.url.IvyAuthenticator
> 11
> WARNING:
> Use --illegal-access=warn to enable warnings of further illegal reflective
> access operations
> 12
> WARNING:
> All illegal access operations will be denied in a future release
> 13
> Scalastyle
> checks failed at following occurrences:
> 14
> 
> [error] (*:update) sbt.ResolveException: unresolved dependency:
> com.typesafe#sbt-mima-plugin;0.3.0: not found
> 15
> ##[error]Process
> completed with exit code 1.
>

I'm not able to reproduce the failure on my local environment, but seems
all the open PRs are failing on this check.

Thanks,

Xingbo

Re: [PMCs] Any project news or announcements this week?

[Sally Khudairi]

Thank you, Xingbo! I'll add you to the list 

- - -
Vice President Marketing & Publicity
Vice President Sponsor Relations
The Apache Software Foundation

Tel +1 617 921 8656 | s...@apache.org


On Tue, Oct 22, 2019, at 03:49, Xingbo Jiang wrote:
> I'm working with Wenchen to make a release candidate, but haven't been 
> successful with our release script. I think we shall be able to make a Spark 
> 3.0 preview release this week, though.
> 
> cc @Sally
> 
> Cheers,
> 
> Xingbo
> 
> Sean Owen  于2019年10月20日周日 下午11:04写道：
>> I wonder if we are likely to have a Spark 3.0 preview release this
>>  week? no rush, but if we do, let's CC Sally to maybe mention at
>>  ApacheCon.
>> 
>>  -- Forwarded message -
>>  From: Sally Khudairi 
>>  Date: Sun, Oct 20, 2019 at 4:00 PM
>>  Subject: [PMCs] Any project news or announcements this week?
>>  To: ASF Marketing & Publicity 
>> 
>> 
>>  Hello Apache PMCs!
>> 
>>  With ApacheCon taking place in Berlin this week, some attending
>>  journalists are interested in covering any project news.
>> 
>>  If your PMC has any newsworthy releases, milestones, etc., taking
>>  place over the next few days, do let me know. It would also be good to
>>  know if you'll be at the conference as well.
>> 
>>  Many thanks,
>>  Sally
>> 
>>  - - -
>>  Vice President Marketing & Publicity
>>  Vice President Sponsor Relations
>>  The Apache Software Foundation
>> 
>>  Tel +1 617 921 8656 | s...@apache.org
>> 
>>  -
>>  To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> 

Re: [PMCs] Any project news or announcements this week?

[Xingbo Jiang]

I'm working with Wenchen to make a release candidate, but haven't been
successful with our release script. I think we shall be able to make a
Spark 3.0 preview release this week, though.

cc @Sally

Cheers,

Xingbo

Sean Owen  于2019年10月20日周日 下午11:04写道：

> I wonder if we are likely to have a Spark 3.0 preview release this
> week? no rush, but if we do, let's CC Sally to maybe mention at
> ApacheCon.
>
> -- Forwarded message -
> From: Sally Khudairi 
> Date: Sun, Oct 20, 2019 at 4:00 PM
> Subject: [PMCs] Any project news or announcements this week?
> To: ASF Marketing & Publicity 
>
>
> Hello Apache PMCs!
>
> With ApacheCon taking place in Berlin this week, some attending
> journalists are interested in covering any project news.
>
> If your PMC has any newsworthy releases, milestones, etc., taking
> place over the next few days, do let me know. It would also be good to
> know if you'll be at the conference as well.
>
> Many thanks,
> Sally
>
> - - -
> Vice President Marketing & Publicity
> Vice President Sponsor Relations
> The Apache Software Foundation
>
> Tel +1 617 921 8656 | s...@apache.org
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>