Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14690
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83518291
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -626,6 +627,40 @@ private[spark] class HiveExternalCatalog(conf:
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83517834
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ListingFileCatalog.scala
---
@@ -17,32 +17,26 @@
package org.apache.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83356659
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -626,6 +627,40 @@ private[spark] class HiveExternalCatalog(con
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83355030
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala ---
@@ -34,7 +34,7 @@ import org.apache.spark.util.Utils
// The data whe
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83352979
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -626,6 +627,40 @@ private[spark] class HiveExternalCatalog(conf:
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83352838
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -626,6 +627,40 @@ private[spark] class HiveExternalCatalog(con
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83349072
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala ---
@@ -34,7 +34,7 @@ import org.apache.spark.util.Utils
// The data whe
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83348167
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val METR
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83348200
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val METR
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83348022
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83347741
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val METR
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83347636
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83347251
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -60,3 +60,32 @@ object CodegenMetrics extends Source {
val
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83342839
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala ---
@@ -34,7 +34,7 @@ import org.apache.spark.util.Utils
// The data where
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83341181
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83340821
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2602,7 +2602,7 @@ class Dataset[T] private[sql](
* @since 2.0.0
*/
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83340736
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2602,7 +2602,7 @@ class Dataset[T] private[sql](
* @since 2.0.0
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83340504
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -477,6 +478,15 @@ class InMemoryCatalog(
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83325529
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,19 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83325088
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed t
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83318096
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83302513
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -230,13 +230,21 @@ case class FileSourceScanExec(
va
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83141382
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83131630
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -199,59 +197,30 @@ private[hive] class
HiveMetastoreCatalog(sp
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83128973
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -199,59 +197,30 @@ private[hive] class
HiveMetastoreCatalog(spar
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83115827
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -199,59 +197,30 @@ private[hive] class
HiveMetastoreCatalog(sp
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83106495
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -55,10 +55,16 @@ class TableFileCatalog(
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83086945
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed t
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83085289
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83085625
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Ap
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83078271
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -55,10 +55,16 @@ class TableFileCatalog(
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83065199
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83071827
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83067223
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apa
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83065354
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,6 +617,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83072226
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83071819
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -55,10 +55,16 @@ class TableFileCatalog(
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83061877
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,19 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r83036315
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed t
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82902172
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82900810
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82875191
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82850487
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82715947
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82715954
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -477,6 +478,15 @@ class InMemoryCatalog(
cat
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82713318
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82713068
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82712965
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -477,6 +478,15 @@ class InMemoryCatalog(
c
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82708580
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82708474
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SessionFileCatalog.scala
---
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Ap
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82709034
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -619,6 +620,44 @@ private[spark] class HiveExternalCatalog(conf:
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82707347
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -477,6 +478,15 @@ class InMemoryCatalog(
cat
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82707709
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r82707684
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -225,13 +225,16 @@ case class FileSourceScanExec(
}
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77428539
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/fileSourceInterfaces.scala
---
@@ -346,11 +340,30 @@ trait FileCatalog {
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77428351
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apac
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426981
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2531,6 +2531,8 @@ class Dataset[T] private[sql](
*/
def inputFiles
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426759
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/fileSourceInterfaces.scala
---
@@ -346,11 +340,30 @@ trait FileCatalog {
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426633
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Ap
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426636
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apac
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426535
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -79,8 +79,16 @@ object FileSourceStrategy extends
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77425878
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Ap
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77425776
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -79,8 +79,16 @@ object FileSourceStrategy exten
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77424735
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -184,7 +184,7 @@ case class FileSourceScanExec(
"Bat
Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77423958
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -184,7 +184,7 @@ case class FileSourceScanExec(
"B
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77422471
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apa
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419498
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apac
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419464
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -79,8 +79,16 @@ object FileSourceStrategy extends
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419510
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/fileSourceInterfaces.scala
---
@@ -346,11 +340,30 @@ trait FileCatalog {
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419448
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -184,7 +184,7 @@ case class FileSourceScanExec(
"Bat
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419496
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apac
Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77419455
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2531,6 +2531,8 @@ class Dataset[T] private[sql](
*/
def inputFiles:
GitHub user mallman opened a pull request:
https://github.com/apache/spark/pull/14690
[SPARK-16980][SQL] Load only catalog table partition metadata required to
answer a query
(This PR addresses https://issues.apache.org/jira/browse/SPARK-16980.)
(N.B. I'm submitting this PR
74 matches
Mail list logo
