deshanxiao created SPARK-25120:
--
Summary: EventLogListener may miss driver
SparkListenerBlockManagerAdded event
Key: SPARK-25120
URL: https://issues.apache.org/jira/browse/SPARK-25120
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-25120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580670#comment-16580670
]
deshanxiao commented on SPARK-25120:
Sure, I find the tab "Executors" in HistorySever sometimes miss
deshanxiao created SPARK-25100:
--
Summary: Using KryoSerializer and setting registrationRequired
true can lead job failed
Key: SPARK-25100
URL: https://issues.apache.org/jira/browse/SPARK-25100
Project:
deshanxiao created SPARK-26457:
--
Summary: Show hadoop configurations in HistoryServer environment
tab
Key: SPARK-26457
URL: https://issues.apache.org/jira/browse/SPARK-26457
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26333:
---
Comment: was deleted
(was: [~vanzin] No, I am not running as root.)
> FsHistoryProviderSuite
[
https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718337#comment-16718337
]
deshanxiao commented on SPARK-26333:
[~vanzin] No, I am not running as root.
>
[
https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718385#comment-16718385
]
deshanxiao commented on SPARK-26333:
[~vanzin] Yes, you are right! Thank you very much! But why
deshanxiao created SPARK-26333:
--
Summary: FsHistoryProviderSuite failed because setReadable doesn't
work in RedHat
Key: SPARK-26333
URL: https://issues.apache.org/jira/browse/SPARK-26333
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739083#comment-16739083
]
deshanxiao commented on SPARK-26570:
[~hyukjin.kwon] OK, I will try it. Thank you!
> Out of memory
[
https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735707#comment-16735707
]
deshanxiao commented on SPARK-26457:
[~planga82]
Hi, thanks for your reply! I know that yarn
[
https://issues.apache.org/jira/browse/SPARK-26528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26528:
---
Priority: Minor (was: Major)
> FsHistoryProviderSuite failed in IDEA because not exist
deshanxiao created SPARK-26528:
--
Summary: FsHistoryProviderSuite failed in IDEA because not exist
"spark.testing" property
Key: SPARK-26528
URL: https://issues.apache.org/jira/browse/SPARK-26528
[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737061#comment-16737061
]
deshanxiao commented on SPARK-26570:
!screenshot-1.png!
> Out of memory when InMemoryFileIndex
[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26570:
---
Attachment: screenshot-1.png
> Out of memory when InMemoryFileIndex bulkListLeafFiles
>
deshanxiao created SPARK-26570:
--
Summary: Out of memory when InMemoryFileIndex bulkListLeafFiles
Key: SPARK-26570
URL: https://issues.apache.org/jira/browse/SPARK-26570
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26570:
---
Description: The *bulkListLeafFiles* will collect all filestatus in memory
for every query which
[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26570:
---
Description: The *bulkListLeafFiles* will collect all filestatus in memory
for every query which
[
https://issues.apache.org/jira/browse/SPARK-26954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26954:
---
Description:
Yarn attemps the failed App depending on YarnRMClient#unregister. However, some
deshanxiao created SPARK-26954:
--
Summary: Do not attemp when user code throws exception
Key: SPARK-26954
URL: https://issues.apache.org/jira/browse/SPARK-26954
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-26954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-26954:
---
Description:
Yarn attemps the failed App depending on YarnRMClient#unregister. However, some
deshanxiao created SPARK-27171:
--
Summary: Support Full-Partiton limit in the first scan
Key: SPARK-27171
URL: https://issues.apache.org/jira/browse/SPARK-27171
Project: Spark
Issue Type:
deshanxiao created SPARK-26714:
--
Summary: The job whose partiton num is zero not shown in WebUI
Key: SPARK-26714
URL: https://issues.apache.org/jira/browse/SPARK-26714
Project: Spark
Issue
deshanxiao created SPARK-28944:
--
Summary: Expose peak memory of executor in metrics for parameter
tuning
Key: SPARK-28944
URL: https://issues.apache.org/jira/browse/SPARK-28944
Project: Spark
deshanxiao created SPARK-28987:
--
Summary: DiskBlockManager#createTempShuffleBlock should skip
directory which is read-only
Key: SPARK-28987
URL: https://issues.apache.org/jira/browse/SPARK-28987
deshanxiao created SPARK-28658:
--
Summary: Yarn FinalStatus is always "success" in yarn-client mode
Key: SPARK-28658
URL: https://issues.apache.org/jira/browse/SPARK-28658
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-28658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-28658:
---
Description:
In yarn-client mode, the finalStatus of application will always be success
because
deshanxiao created SPARK-29711:
--
Summary: Dynamic adjust spark sql class log level in beeline
Key: SPARK-29711
URL: https://issues.apache.org/jira/browse/SPARK-29711
Project: Spark
Issue Type:
deshanxiao created SPARK-30123:
--
Summary: PartitionPruning should consider more case
Key: SPARK-30123
URL: https://issues.apache.org/jira/browse/SPARK-30123
Project: Spark
Issue Type:
deshanxiao created SPARK-30106:
--
Summary: DynamicPartitionPruningSuite#"no predicate on the
dimension table" is not be tested
Key: SPARK-30106
URL: https://issues.apache.org/jira/browse/SPARK-30106
[
https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983536#comment-16983536
]
deshanxiao commented on SPARK-27780:
I can't argee it more. Add shuffle service version is very
deshanxiao created SPARK-30073:
--
Summary: HistoryPage render "count" cost too much time
Key: SPARK-30073
URL: https://issues.apache.org/jira/browse/SPARK-30073
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-30073:
---
Description:
{code:java}
"qtp1010584177-537" #537 daemon prio=5 os_prio=0 tid=0x7f2734185000
[
https://issues.apache.org/jira/browse/SPARK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-30073:
---
Environment: (was:
{code:java}
"qtp1010584177-537" #537 daemon prio=5 os_prio=0
[
https://issues.apache.org/jira/browse/SPARK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-30073:
---
Affects Version/s: (was: 3.0.0)
2.3.2
> HistoryPage render "count" cost
[
https://issues.apache.org/jira/browse/SPARK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984786#comment-16984786
]
deshanxiao commented on SPARK-30073:
[~kabhwan]
Sorry, I have changed it to spark2.3.2. Thank you!
[
https://issues.apache.org/jira/browse/SPARK-30883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-30883:
---
Environment: The java api *setWritable,setReadable and setExecutable*
dosen't work well when the
deshanxiao created SPARK-30883:
--
Summary: Tests that use setWritable,setReadable and setExecutable
should be cancel when user is root
Key: SPARK-30883
URL: https://issues.apache.org/jira/browse/SPARK-30883
[
https://issues.apache.org/jira/browse/SPARK-30883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-30883:
---
Environment: The java api *setWritable,setReadable and setExecutable*
dosen't work well because
[
https://issues.apache.org/jira/browse/SPARK-31112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31112:
---
Description: Now, we use HiveClientImpl to access hive metastore. However,
a long running rpc in
[
https://issues.apache.org/jira/browse/SPARK-31112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31112:
---
Description: Now, we use HiveClientImpl to access hive metastore. However,
a long running rpc in
deshanxiao created SPARK-31112:
--
Summary: Use multiple extrenal catalog to speed up metastore access
Key: SPARK-31112
URL: https://issues.apache.org/jira/browse/SPARK-31112
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-31112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31112:
---
Description:
Now, we use HiveClientImpl to access hive metastore. However, a long running
rpc in
[
https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31140:
---
Description:
RDD.sample use the function of *filter* to pick up the data we need. It means
that if
deshanxiao created SPARK-31140:
--
Summary: Support Quick sample in RDD
Key: SPARK-31140
URL: https://issues.apache.org/jira/browse/SPARK-31140
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31140:
---
Description:
RDD.sample use the function of *filter* to pick up the data we need. It means
that if
[
https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059916#comment-17059916
]
deshanxiao commented on SPARK-31140:
Sure you are right. I just suggest that if we could add a new
[
https://issues.apache.org/jira/browse/SPARK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059908#comment-17059908
]
deshanxiao commented on SPARK-31140:
[~viirya] Thanks for your comment! It mean that we can
deshanxiao created SPARK-39916:
--
Summary: Merge SchemaUtils from mlib to SQL
Key: SPARK-39916
URL: https://issues.apache.org/jira/browse/SPARK-39916
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-39916:
---
Description: Today we have two SchemaUtils: SQL SchemaUtils and mllib
SchemaUtils. the SchemaUtils
[
https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-39916:
---
Description: Today we have two SchemaUtils: SQL SchemaUtils and mllib
SchemaUtils. the SchemaUtils
deshanxiao created SPARK-40192:
--
Summary: Remove redundant groupby
Key: SPARK-40192
URL: https://issues.apache.org/jira/browse/SPARK-40192
Project: Spark
Issue Type: Improvement
deshanxiao created SPARK-40472:
--
Summary: Improve pyspark.sql.function example experience
Key: SPARK-40472
URL: https://issues.apache.org/jira/browse/SPARK-40472
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-40472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40472:
---
Description:
There are many exanple in pyspark.sql.function:
{code:java}
Examples
[
https://issues.apache.org/jira/browse/SPARK-40472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao resolved SPARK-40472.
Resolution: Fixed
> Improve pyspark.sql.function example experience
>
[
https://issues.apache.org/jira/browse/SPARK-40472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606879#comment-17606879
]
deshanxiao commented on SPARK-40472:
[~hyukjin.kwon] OK, thanks~
> Improve pyspark.sql.function
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40103:
---
Summary: Support read/write.csv() in SparkR (was: Support read.csv() in
SparkR)
> Support
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40103:
---
Summary: Support read.csv() in SparkR (was: Support read.csv in SparkR)
> Support read.csv() in
[
https://issues.apache.org/jira/browse/SPARK-39934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580183#comment-17580183
]
deshanxiao commented on SPARK-39934:
[~hyukjin.kwon] I have confirmed the code below, the takeRDD
deshanxiao created SPARK-40103:
--
Summary: Support read.csv in SparkR
Key: SPARK-40103
URL: https://issues.apache.org/jira/browse/SPARK-40103
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40103:
---
Description:
Today, all major languages support the DataFrameReader.csv API, only R is
missing. we
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40103:
---
Issue Type: New Feature (was: Improvement)
> Support read/write.csv() in SparkR
>
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-40103:
---
Description:
Today, almost languages support the DataFrameReader.csv API, only R is missing.
we
[
https://issues.apache.org/jira/browse/SPARK-39934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575221#comment-17575221
]
deshanxiao commented on SPARK-39934:
[~hyukjin.kwon] Hi, Maybe there is something wrong with my
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580623#comment-17580623
]
deshanxiao edited comment on SPARK-40103 at 8/17/22 7:23 AM:
-
Yes read.csv,
[
https://issues.apache.org/jira/browse/SPARK-40103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580623#comment-17580623
]
deshanxiao commented on SPARK-40103:
Yes read.csv, read.csv2 have benn used in R utils packages.
>
[
https://issues.apache.org/jira/browse/SPARK-39934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-39934:
---
Description: The api of SparkR:::takeRDD retrieves the result one partition
per round. We can
deshanxiao created SPARK-39934:
--
Summary: takeRDD in R is slow
Key: SPARK-39934
URL: https://issues.apache.org/jira/browse/SPARK-39934
Project: Spark
Issue Type: Improvement
67 matches
Mail list logo
