[
https://issues.apache.org/jira/browse/SPARK-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718580#comment-15718580
]
Li Yuanjian edited comment on SPARK-18700 at 12/9/16 4:47 AM:
--
Give a PR for
[
https://issues.apache.org/jira/browse/SPARK-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-18700:
Description:
In our spark sql platform, each query use same HiveContext and independent
Li Yuanjian created SPARK-18700:
---
Summary: getCached in HiveMetastoreCatalog not thread safe cause
driver OOM
Key: SPARK-18700
URL: https://issues.apache.org/jira/browse/SPARK-18700
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718580#comment-15718580
]
Li Yuanjian commented on SPARK-18700:
-
I'll add PR for this soon, add ReadWriteLock for each table's
[
https://issues.apache.org/jira/browse/SPARK-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-18700:
Affects Version/s: 2.1.1
> getCached in HiveMetastoreCatalog not thread safe cause driver OOM
>
[
https://issues.apache.org/jira/browse/SPARK-20408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-20408:
Summary: Get glob path in parallel to reduce resolve relation time (was:
Get glob path in
Li Yuanjian created SPARK-20408:
---
Summary: Get glob path in parallel to boost resolve relation time
Key: SPARK-20408
URL: https://issues.apache.org/jira/browse/SPARK-20408
Project: Spark
Issue
Li Yuanjian created SPARK-21560:
---
Summary: Add hold mode for the LiveListenerBus
Key: SPARK-21560
URL: https://issues.apache.org/jira/browse/SPARK-21560
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-21435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089442#comment-16089442
]
Li Yuanjian commented on SPARK-21435:
-
[~sowen] I tested the patch in our scenario and add a UT, I
Li Yuanjian created SPARK-21435:
---
Summary: Empty files should be skipped while write to file
Key: SPARK-21435
URL: https://issues.apache.org/jira/browse/SPARK-21435
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-21560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106030#comment-16106030
]
Li Yuanjian commented on SPARK-21560:
-
Something wrong with the sync between PR and JIRA, the PR
[
https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119265#comment-16119265
]
Li Yuanjian commented on SPARK-18838:
-
I'm facing the same problem with [~milesc]
{quote}
We do not
Li Yuanjian created SPARK-22074:
---
Summary: Task killed by other attempt task should not be
resubmitted
Key: SPARK-22074
URL: https://issues.apache.org/jira/browse/SPARK-22074
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-22074:
Description:
When a task killed by other task attempt, the task still resubmitted while its
[
https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180652#comment-16180652
]
Li Yuanjian commented on SPARK-22074:
-
Hi [~jerryshao], thanks for you comment.
In my scenario, the
[
https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181943#comment-16181943
]
Li Yuanjian commented on SPARK-22074:
-
Hi [~jerryshao] saisai, the 66.0 resubmitted because of its
[
https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181943#comment-16181943
]
Li Yuanjian edited comment on SPARK-22074 at 9/27/17 4:46 AM:
--
Hi
[
https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182095#comment-16182095
]
Li Yuanjian commented on SPARK-22074:
-
Yes, that's right.
> Task killed by other attempt task should
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265109#comment-16265109
]
Li Yuanjian commented on SPARK-2926:
Yes, only the reduce stage. You're right, I shouldn't only pay
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265030#comment-16265030
]
Li Yuanjian commented on SPARK-2926:
[~jerryshao], thanks a lot for your advise and reply.
{quote}
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-2926:
---
Attachment: Spark Shuffle Test Report on Spark2.x.pdf
[~jerryshao] Hi saisai, thanks for your advise,
Li Yuanjian created SPARK-22546:
---
Summary: Allow users to update the dataType of a column
Key: SPARK-22546
URL: https://issues.apache.org/jira/browse/SPARK-22546
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251398#comment-16251398
]
Li Yuanjian commented on SPARK-2926:
During our work of migrating some old Hadoop job to Spark, I
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-2926:
---
Comment: was deleted
(was: The follow up work for SortShuffleReader in current master branch, detail
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-2926:
---
Attachment: SortBasedShuffleReader on Spark 2.x.pdf
The follow up work for SortShuffleReader in
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251398#comment-16251398
]
Li Yuanjian edited comment on SPARK-2926 at 11/14/17 1:54 PM:
--
During our
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251398#comment-16251398
]
Li Yuanjian edited comment on SPARK-2926 at 11/14/17 1:53 PM:
--
During our
[
https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251457#comment-16251457
]
Li Yuanjian commented on SPARK-2926:
I just giving a preview PR above, I'll collect more suggestions
[
https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245556#comment-16245556
]
Li Yuanjian commented on SPARK-20928:
-
Our team discuss on the design sketch in detail, we have some
Li Yuanjian created SPARK-22753:
---
Summary: Get rid of dataSource.writeAndRead
Key: SPARK-22753
URL: https://issues.apache.org/jira/browse/SPARK-22753
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470067#comment-16470067
]
Li Yuanjian commented on SPARK-24036:
-
I agree with the division about the kinds of tasks, that's
[
https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-24235:
Description:
[
https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-24235:
Description:
[
https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-24235:
Description:
[
https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469830#comment-16469830
]
Li Yuanjian commented on SPARK-24036:
-
Hi [~joseph.torres]
Thanks for cc me, looks great!
My doc
[
https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469830#comment-16469830
]
Li Yuanjian edited comment on SPARK-24036 at 5/10/18 2:32 AM:
--
Hi
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23128:
Comment: was deleted
(was: I collected some user cases and performance improve effect during Baidu
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471811#comment-16471811
]
Li Yuanjian commented on SPARK-23128:
-
I collected some user cases and performance improve effect
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471812#comment-16471812
]
Li Yuanjian commented on SPARK-23128:
-
I collected some user cases and performance improve effect
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23128:
Attachment: AdaptiveExecutioninBaidu.pdf
> A new approach to do adaptive execution in Spark SQL
>
Li Yuanjian created SPARK-24304:
---
Summary: Scheduler changes for continuous processing shuffle
support
Key: SPARK-24304
URL: https://issues.apache.org/jira/browse/SPARK-24304
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478406#comment-16478406
]
Li Yuanjian commented on SPARK-24293:
-
{quote}
doing the map side combine manually with SQL
[
https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506285#comment-16506285
]
Li Yuanjian commented on SPARK-24499:
-
No problem, thanks for ping me, our pleasure.
We'll collect
[
https://issues.apache.org/jira/browse/SPARK-24183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504614#comment-16504614
]
Li Yuanjian commented on SPARK-24183:
-
Hi [~joseph.torres], I notice currently we already have
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504171#comment-16504171
]
Li Yuanjian commented on SPARK-24375:
-
Got it, great thanks for your detailed explanation.
> Design
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503486#comment-16503486
]
Li Yuanjian commented on SPARK-24375:
-
Hi [~cloud_fan] and [~jiangxb1987], just I tiny question
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503486#comment-16503486
]
Li Yuanjian edited comment on SPARK-24375 at 6/6/18 3:55 PM:
-
Hi
[
https://issues.apache.org/jira/browse/SPARK-24210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500383#comment-16500383
]
Li Yuanjian commented on SPARK-24210:
-
I think it maybe not a bug.
#KO: returns r1 and
[
https://issues.apache.org/jira/browse/SPARK-24210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500383#comment-16500383
]
Li Yuanjian edited comment on SPARK-24210 at 6/4/18 3:34 PM:
-
I think it
[
https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523064#comment-16523064
]
Li Yuanjian commented on SPARK-24630:
-
cc [~zsxwing] and [~tdas]
We have some practice over
Li Yuanjian created SPARK-24665:
---
Summary: Add SQLConf in PySpark to manage all sql configs
Key: SPARK-24665
URL: https://issues.apache.org/jira/browse/SPARK-24665
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453591#comment-16453591
]
Li Yuanjian commented on SPARK-21661:
-
[~hyukjin.kwon] Hi Hyukjin, do you think we should backport
[
https://issues.apache.org/jira/browse/SPARK-24108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian closed SPARK-24108.
---
Duplicated submit for
[SPARK-24107|https://issues.apache.org/jira/browse/SPARK-24107], just close it.
Li Yuanjian created SPARK-22956:
---
Summary: Union Stream Failover Cause `IllegalStateException`
Key: SPARK-22956
URL: https://issues.apache.org/jira/browse/SPARK-22956
Project: Spark
Issue
Li Yuanjian created SPARK-24989:
---
Summary: BlockFetcher should retry while getting
OutOfDirectMemoryError
Key: SPARK-24989
URL: https://issues.apache.org/jira/browse/SPARK-24989
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-24989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-24989:
Attachment: FailedStage.png
> BlockFetcher should retry while getting OutOfDirectMemoryError
>
[
https://issues.apache.org/jira/browse/SPARK-24989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian resolved SPARK-24989.
-
Resolution: Not A Problem
The param `spark.reducer.maxBlocksInFlightPerAddress` added in
Li Yuanjian created SPARK-25077:
---
Summary: Delete unused variable in WindowExec
Key: SPARK-25077
URL: https://issues.apache.org/jira/browse/SPARK-25077
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-25100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-25100:
Description:
When spark.serializer is `org.apache.spark.serializer.KryoSerializer` and
[
https://issues.apache.org/jira/browse/SPARK-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-25104:
Description:
With code changes in
[
https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587326#comment-16587326
]
Li Yuanjian commented on SPARK-25050:
-
Has more than tow types in avro union types been supported in
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587348#comment-16587348
]
Li Yuanjian commented on SPARK-23128:
-
Thanks for your comment [~xinyao].
{quote}
If I understand
[
https://issues.apache.org/jira/browse/SPARK-25072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584705#comment-16584705
]
Li Yuanjian commented on SPARK-25072:
-
Interesting issue, but maybe this only in PySpark, Scala Row
[
https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551638#comment-16551638
]
Li Yuanjian commented on SPARK-23128:
-
[~tgraves] Thanks for your comment, as far as I know
[
https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544453#comment-16544453
]
Li Yuanjian commented on SPARK-24295:
-
Could you give more detailed information about how the
[
https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547873#comment-16547873
]
Li Yuanjian commented on SPARK-24295:
-
Thanks for your detailed explain.
You can check this:
[
https://issues.apache.org/jira/browse/SPARK-24340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551687#comment-16551687
]
Li Yuanjian commented on SPARK-24340:
-
cc [~jiangxb1987] I think this was resolved by your pr 21390,
[
https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536107#comment-16536107
]
Li Yuanjian commented on SPARK-24755:
-
No problem, thanks [~hthuynh2].
Thanks [~mridulm80] for
[
https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23811:
Description:
This is a bug caused by abnormal scenario describe below:
# ShuffleMapTask 1.0
[
https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418434#comment-16418434
]
Li Yuanjian commented on SPARK-23811:
-
The scenario can be reproduced by below test case added in
[
https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23811:
Attachment: 1.png
> Same tasks' FetchFailed event comes before Success will cause child stage
>
Li Yuanjian created SPARK-23811:
---
Summary: Same tasks' FetchFailed event comes before Success will
cause child stage never succeed
Key: SPARK-23811
URL: https://issues.apache.org/jira/browse/SPARK-23811
[
https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23811:
Attachment: 2.png
> Same tasks' FetchFailed event comes before Success will cause child stage
>
[
https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23811:
Summary: FetchFailed comes before Success of same task will cause child
stage never succeed (was:
[
https://issues.apache.org/jira/browse/SPARK-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-23533:
Summary: Add support for changing ContinuousDataReader's startOffset (was:
Add support for
Li Yuanjian created SPARK-23533:
---
Summary: Add support for changing ContinousDataReader's startOffset
Key: SPARK-23533
URL: https://issues.apache.org/jira/browse/SPARK-23533
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454070#comment-16454070
]
Li Yuanjian commented on SPARK-21661:
-
Got it.
> SparkSQL can't merge load table from Hadoop
>
[
https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631300#comment-16631300
]
Li Yuanjian commented on SPARK-22565:
-
Also cc [~zsxwing] [~tdas], we are translating the design doc
[
https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-22565:
Attachment: screenshot-1.png
> Session-based windowing
> ---
>
>
[
https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631298#comment-16631298
]
Li Yuanjian edited comment on SPARK-22565 at 9/28/18 3:23 AM:
--
Thanks for
[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631398#comment-16631398
]
Li Yuanjian commented on SPARK-10816:
-
Great thanks for [~kabhwan] notice me, just linked
[
https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631298#comment-16631298
]
Li Yuanjian commented on SPARK-22565:
-
Thanks for reporting this. Actually we also met this problem
[
https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631370#comment-16631370
]
Li Yuanjian commented on SPARK-22565:
-
[~kabhwan]
Great thanks for noticing me, sorry for only
[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646177#comment-16646177
]
Li Yuanjian commented on SPARK-10816:
-
Thanks [~zsxwing] for your comment and discussion, great
[
https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625061#comment-16625061
]
Li Yuanjian commented on SPARK-24499:
-
[~smilegator] Sorry for the late reply for this, we'll give
[
https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627593#comment-16627593
]
Li Yuanjian commented on SPARK-25527:
-
{quote}
There are no Tasks waiting for completion, and the
[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Yuanjian updated SPARK-10816:
Attachment: Session Window Support For Structure Streaming.pdf
> EventTime based sessionization
>
[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631751#comment-16631751
]
Li Yuanjian commented on SPARK-10816:
-
Design doc:
[
https://issues.apache.org/jira/browse/SPARK-25426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617587#comment-16617587
]
Li Yuanjian commented on SPARK-25426:
-
Resolved by https://github.com/apache/spark/pull/22417.
>
89 matches
Mail list logo
