[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353325#comment-16353325
]
Wenchen Fan commented on SPARK-23304:
-
I didn't follow the discussion here closely, but FYI, the
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350440#comment-16350440
]
Thomas Graves commented on SPARK-23304:
---
ok so I guess by that logic then the coalesce won't every
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350423#comment-16350423
]
Thomas Graves commented on SPARK-23304:
---
it doesn't look like sql("xyz").rdd.partitions.length
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350428#comment-16350428
]
Thomas Graves commented on SPARK-23304:
---
well I guess that give you end # of partitions and not the
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349555#comment-16349555
]
Thomas Graves commented on SPARK-23304:
---
I don't have any hive tables backed by parquet to compare
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349511#comment-16349511
]
Xiao Li commented on SPARK-23304:
-
Based on the new plan, it sounds like the plan is not changed. Could
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349498#comment-16349498
]
Dongjoon Hyun commented on SPARK-23304:
---
I updated the affected version according to the latest
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349480#comment-16349480
]
Thomas Graves commented on SPARK-23304:
---
I just ran the query (show()) and saw the # of partitions.
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349461#comment-16349461
]
Xiao Li commented on SPARK-23304:
-
That is fine.
Obviously, at least, we need to submit a PR to document
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349446#comment-16349446
]
Thomas Graves commented on SPARK-23304:
---
It still seems like a bug to me since the coalesce isn't
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349444#comment-16349444
]
Thomas Graves commented on SPARK-23304:
---
[~smilegator] just to make sure you saw my comment above,
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349436#comment-16349436
]
Xiao Li commented on SPARK-23304:
-
In this release, we also made a change in the default of another
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349430#comment-16349430
]
Thomas Graves commented on SPARK-23304:
---
I filed Jira
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349369#comment-16349369
]
Thomas Graves commented on SPARK-23304:
---
Note I've removed some of the columns from the output, if
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349367#comment-16349367
]
Thomas Graves commented on SPARK-23304:
---
ok I've attached 2 files one with spark 2.3 and one with
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349346#comment-16349346
]
Sameer Agarwal commented on SPARK-23304:
Also, is there a JIRA/repro for the caching issue you
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349340#comment-16349340
]
Xiao Li commented on SPARK-23304:
-
I do not think our native ORC reader respects
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349339#comment-16349339
]
Xiao Li commented on SPARK-23304:
-
Hi, [~tgraves], could you change the two SQLConf
`spark.sql.orc.impl`
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349270#comment-16349270
]
Thomas Graves commented on SPARK-23304:
---
so with the new ORC code is there anyway to control the #
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349227#comment-16349227
]
Thomas Graves commented on SPARK-23304:
---
Ok, I just realized what you are getting at, I tried on
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349220#comment-16349220
]
Thomas Graves commented on SPARK-23304:
---
If it helps , spark 2.3 # partitions is 317531 and spark
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349212#comment-16349212
]
Thomas Graves commented on SPARK-23304:
---
yes there are difference in the # of partitions between
[
https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349147#comment-16349147
]
Sameer Agarwal commented on SPARK-23304:
[~tgraves] just to rule out the obvious, was there a
23 matches
Mail list logo
