[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854534#comment-15854534
]
Ian commented on SPARK-19462:
-
I appears that the state mutating of newPartitioning of
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853408#comment-15853408
]
Ian commented on SPARK-19462:
-
Note that when spark.sql.adaptive.enabled is set "false" , the exception
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Description:
property spark.sql.adaptive.enabled needs to be set "true" for the issue to be
reproduced.
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853188#comment-15853188
]
Ian edited comment on SPARK-19462 at 2/5/17 10:33 AM:
--
In fact, the "Exchange not
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Summary: when spark.sql.adaptive.enabled is enabled, DF is not resilient to
node/container failure (was: when
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Summary: when spark.sql.adaptive.enabled is enabled, DF is not resilient to
node container failure (was: when
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853188#comment-15853188
]
Ian edited comment on SPARK-19462 at 2/5/17 10:28 AM:
--
In fact, the "Exchange not
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Description:
property spark.sql.adaptive.enabled needs to be set "true"
reproducible steps using spark-shell
0.
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Comment: was deleted
(was: When the RDD lineage contains ShuffledRowRDD, the above mentioned
behavior can be
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Description:
property spark.sql.adaptive.enabled needs to be set "true"
reproducible steps using spark-shell
0.
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Summary: when spark.sql.adaptive.enabled is enabled, RDD is not resilient
to node container failure (was: when
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853188#comment-15853188
]
Ian commented on SPARK-19462:
-
In fact, the "Exchange not implemented for UnknownPartitioning(1)" can be
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853184#comment-15853184
]
Ian edited comment on SPARK-19462 at 2/5/17 10:14 AM:
--
When the RDD lineage contains
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853184#comment-15853184
]
Ian commented on SPARK-19462:
-
When the RDD lineage contains ShuffledRowRDD, the above mentioned behavior can
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Description:
property spark.sql.adaptive.enabled needs to be set "true"
reproducible steps using spark-shell
0.
[
https://issues.apache.org/jira/browse/SPARK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-19462:
Description:
property spark.sql.adaptive.enabled needs to be set "true"
reproducible steps using spark-shell
0.
Ian created SPARK-19462:
---
Summary: when spark.sql.adaptive.enabled is enabled RDD is not
resilient to node container failure
Key: SPARK-19462
URL: https://issues.apache.org/jira/browse/SPARK-19462
Project:
[
https://issues.apache.org/jira/browse/SPARK-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566439#comment-15566439
]
Ian commented on SPARK-11784:
-
Yes, I meant TimestampType filter pushdown
> enable Timestamp filter pushdown
[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530
]
Ian commented on SPARK-11153:
-
Cheng,
Can we revisit SPARK-11784?
CC [~markhamstra]
> Turns off Parquet
[
https://issues.apache.org/jira/browse/SPARK-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308660#comment-15308660
]
Ian commented on SPARK-6859:
is this one fixed along with SPARK-9876?
> Parquet File Binary column statistics
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194014#comment-15194014
]
Ian edited comment on SPARK-13872 at 3/14/16 7:53 PM:
--
A spark plan illustrating the
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Description:
SortMergeJoin composes its partition/iterator from
org.apache.spark.sql.execution.Sort, which in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194014#comment-15194014
]
Ian commented on SPARK-13872:
-
A spark plan illustrating the scenario was attached.
1. A cartesian is
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Summary: Memory leak in SortMergeOuterJoin (was: Memory leak
SortMergeOuterJoin)
> Memory leak in
[
https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13872:
Attachment: Screen Shot 2016-03-11 at 5.42.32 PM.png
> Memory leak SortMergeOuterJoin
>
Ian created SPARK-13872:
---
Summary: Memory leak SortMergeOuterJoin
Key: SPARK-13872
URL: https://issues.apache.org/jira/browse/SPARK-13872
Project: Spark
Issue Type: Bug
Components: SQL
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183869#comment-15183869
]
Ian edited comment on SPARK-13731 at 3/8/16 7:29 PM:
-
We saw SPARK-9076, which
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184034#comment-15184034
]
Ian edited comment on SPARK-13731 at 3/8/16 2:35 AM:
-
The test case we provided is
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184034#comment-15184034
]
Ian edited comment on SPARK-13731 at 3/8/16 2:35 AM:
-
The test case we provided is
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183869#comment-15183869
]
Ian edited comment on SPARK-13731 at 3/8/16 12:04 AM:
--
The expression in select
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13731:
Description:
We are expecting that arithmetic expression a/b should be:
1. returning NaN if a=0 and b=0
2.
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184034#comment-15184034
]
Ian edited comment on SPARK-13731 at 3/7/16 11:41 PM:
--
The test case we provided is
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184034#comment-15184034
]
Ian commented on SPARK-13731:
-
The test case we provided is using simple arithmetic expression like divisions
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13731:
Description:
We are expecting that arithmetic expression a/b should be:
1. returning NaN if a=0 and b=0
2.
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183869#comment-15183869
]
Ian edited comment on SPARK-13731 at 3/7/16 10:33 PM:
--
The expression in select
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183869#comment-15183869
]
Ian commented on SPARK-13731:
-
The expression in select essentially defined a transformation from data
[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-13731:
Description:
We are expecting arithmetic expression a/b should be:
1. returning NaN if a=0 and b=0
2. returning
Ian created SPARK-13731:
---
Summary: expression evaluation for NaN in select statement
Key: SPARK-13731
URL: https://issues.apache.org/jira/browse/SPARK-13731
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-9876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109507#comment-15109507
]
Ian commented on SPARK-9876:
[~liancheng] do you think we can revisit this now?
cc [~markhamstra]
> Upgrade
[
https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050201#comment-15050201
]
Ian commented on SPARK-12258:
-
I believed this is a regression in 1.6
> Hive Timestamp UDF is binded with
[
https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-12258:
Description:
{code}
test("Timestamp UDF and Null value") {
hiveContext.runSqlHive("CREATE TABLE ts_test (ts
Ian created SPARK-12258:
---
Summary: Hive Timestamp UDF is binded with '1969-12-31
15:59:59.99' for null value
Key: SPARK-12258
URL: https://issues.apache.org/jira/browse/SPARK-12258
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008897#comment-15008897
]
Ian commented on SPARK-11153:
-
Cheng:
Any plan to enable pushdown for timestamps?
Should I open a ticket
Ian created SPARK-11784:
---
Summary: enable Timestamp filter pushdown
Key: SPARK-11784
URL: https://issues.apache.org/jira/browse/SPARK-11784
Project: Spark
Issue Type: Bug
Affects Versions: 1.5.1
[
https://issues.apache.org/jira/browse/SPARK-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-11784:
Component/s: SQL
> enable Timestamp filter pushdown
>
>
> Key:
[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009279#comment-15009279
]
Ian commented on SPARK-11153:
-
SPARK-11784 is now tracking the need of timestamp pushdown.
> Turns off
[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008074#comment-15008074
]
Ian edited comment on SPARK-11153 at 11/17/15 5:18 AM:
---
Hi, Cheng:
How about
[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008074#comment-15008074
]
Ian commented on SPARK-11153:
-
Hi, Cheng:
Filter pushdown for Timestamp type?
It appeared that the stats of
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906801#comment-14906801
]
Ian edited comment on SPARK-10741 at 9/24/15 6:46 PM:
--
Two of my tests failed. The
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906801#comment-14906801
]
Ian commented on SPARK-10741:
-
Two of my tests failed.
The query returns nothing.
{code}
test("test
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906835#comment-14906835
]
Ian commented on SPARK-10741:
-
yup, it works.
The following insert select statement works for non parquet
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906768#comment-14906768
]
Ian commented on SPARK-10741:
-
The org.apache.spark.sql.AnalysisException is fixed, but the write path seemed
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905327#comment-14905327
]
Ian edited comment on SPARK-10741 at 9/23/15 9:36 PM:
--
Yes, going through all rules
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905327#comment-14905327
]
Ian commented on SPARK-10741:
-
Yes, going through all rules when resolve Sort on Aggregate is a correct
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905327#comment-14905327
]
Ian edited comment on SPARK-10741 at 9/23/15 9:45 PM:
--
Yes, going through all rules
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-10741:
Description:
Failed Query with Having Clause
{code}
def testParquetHaving() {
val ddl =
"""CREATE
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-10741:
Description:
Failed Query with Having Clause
{code}
def testParquetHaving() {
val ddl =
"""CREATE
Ian created SPARK-10741:
---
Summary: Hive Query Having/OrderBy against Parquet table is not
working
Key: SPARK-10741
URL: https://issues.apache.org/jira/browse/SPARK-10741
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-10741:
Description:
Failed Query with Having Clause
{code}
def testParquetHaving() {
val ddl =
"""CREATE
[
https://issues.apache.org/jira/browse/SPARK-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian updated SPARK-10741:
Description:
Failed Query with Having Clause
{code}
def testParquetHaving() {
val ddl =
"""CREATE
65 matches
Mail list logo
