Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23208#discussion_r240101369
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,52 +17,49
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23215
Thank you @cloud-fan @viirya @HyukjinKwon .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23215
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23208#discussion_r239454594
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/SupportsBatchWrite.java
---
@@ -25,14 +25,14 @@
import
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23215#discussion_r239423651
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -272,9 +279,13 @@ object
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23215
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23240
Oh, I see. Close this one now.
Please change the title in #23160
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gengliangwang closed the pull request at:
https://github.com/apache/spark/pull/23240
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23240
@shahidki31 @pgandhi999 @srowen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23240
[SPARK-26281][WebUI] Duration column of task table should be executor run
time instead of real duration
## What changes were proposed in this pull request?
In PR https://github.com
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23215
[SPARK-26263][SQL] Throw exception when Partition column value can't be
converted to user specified type
## What changes were proposed in this pull request?
Currently if
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23215
@cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23189
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23189#discussion_r237972852
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -813,14 +813,14 @@ private[spark] class SparkSubmit extends Logging
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23189
@vanzin Ah, I see. Thanks for pointing it out!
But I am now thinking overriding `logError` by calling
`printMessage("Error...")`. What do
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23186#discussion_r237963856
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -345,15 +346,18 @@ object
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23189
@vanzin The `logWarning` call as the other handler below is also not
overridden:
```
case e: NoClassDefFoundError =>
logWarning(s"Failed to load $childM
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23189
cc @vanzin
This is really trivial, but can be helpful to user in certain case.
---
-
To unsubscribe, e-mail
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23189
[SPARK-26235][Core] Change log level for
ClassNotFoundException/NoClassDefFoundError in SparkSubmit to Error
## What changes were proposed in this pull request?
In my local setup, I
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23160
cc @tgravescs @pgandhi999
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23186
@cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23186
[SPARK-26230][SQL]FileIndex: if case sensitive, validate partitions with
original column names
## What changes were proposed in this pull request?
Partition column name is required
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
@gatorsmile thanks! For Jobs don't have associated SQL query, the text
`Associated SQL query` won't show up.
---
--
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23165#discussion_r237523177
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -250,7 +276,13 @@ object
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23165#discussion_r237516228
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -250,7 +276,13 @@ object
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23165#discussion_r237515959
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -250,7 +276,13 @@ object
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23165#discussion_r237513657
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -94,18 +94,34 @@ object
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23165#discussion_r237512471
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -94,18 +94,34 @@ object
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23165
@mgaido91 The user specified schema might not match the full data schema.
For the missing columns, we still need to infer their data types.
I will come up with a solution soon
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23165
@cloud-fan @mgaido91
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23165
[SPARK-26188][SQL] FileIndex: don't infer data types of partition columns
if user specifies schema
## What changes were proposed in this pull request?
This PR is to fix a regre
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21004#discussion_r237092452
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
---
@@ -126,35 +126,32 @@ abstract
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
ping @vanzin . Please review this one :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23125#discussion_r235995724
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -79,6 +79,9 @@ private[ui] class StagePage(parent: StagesTab, store
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23125
[SPARK-26156][WebUI] Revise summary section of stage page
## What changes were proposed in this pull request?
In the summary section of stage page:
![image](https://user
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23125
@srowen @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23098#discussion_r235317414
--- Diff: bin/load-spark-env.cmd ---
@@ -21,37 +21,42 @@ rem This script loads spark-env.cmd if it exists, and
ensures it is only loaded
rem
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
The web UI presents "jobs" tab as the default view. Showing the related SQL
context is helpful, especially for new users.
I think the downside is qu
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23081#discussion_r235071529
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -996,7 +996,7 @@ private[ui] object ApiHelper {
HEADER_EXECUTOR
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23081#discussion_r235065240
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -996,7 +996,7 @@ private[ui] object ApiHelper {
HEADER_EXECUTOR
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23081
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
@srowen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23067
+1.
We can also try overriding `toString` for better output.
@vanzin
---
-
To unsubscribe, e-mail: reviews
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23088
As the task section shows all the tasks, I think maybe it would be better
to show aggregated metrics for all the tasks
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23088#discussion_r235027227
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -238,8 +239,16 @@ private[spark] class AppStatusStore
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23038#discussion_r234728748
--- Diff:
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23065
I tried and it works.
There is a similar warning in UnionRDD.scala, which will cause failure in
Scala 2.11.
---
-
To
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23065
Hi @srowen ,
Could you review and merge https://github.com/srowen/spark/pull/4 ?
I see a lot of warnings as well. We should fix them
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
@vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23068
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23068#discussion_r234402149
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -56,6 +56,11 @@ private[spark] class AppStatusStore
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23068#discussion_r234402077
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -70,6 +70,8 @@ private[spark] class AppStatusListener
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23068
[SPARK-26098][WebUI] Show associated SQL query in Job page
## What changes were proposed in this pull request?
For jobs associated to SQL queries, it would be easier to understand
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23049
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23049
@vanzin I see your point. I will add a link to
https://spark.apache.org/docs/latest/configuration.html. Thanks for the
suggestion.
In my case, I didn't know where to find or edit `
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23049
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang closed the pull request at:
https://github.com/apache/spark/pull/23047
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23049
Hi @vanzin ,
thanks for pointing it out! I have updated the script and PR description.
---
-
To unsubscribe, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23049#discussion_r233902416
--- Diff: bin/load-spark-env.sh ---
@@ -47,8 +47,8 @@ if [ -z "$SPARK_SCALA_VERSION" ]; then
ASSEMBLY_DIR1="${SPARK_HOME}/assemb
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/23049
@srowen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23049
[SPARK-26076][Build] Revise ambiguous error message from load-spark-env.sh
## What changes were proposed in this pull request?
When I try to run scripts (e.g. `./sbin/start-history
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/23047
[SPARK-25883][SQL][MINOR] Override method `prettyName` in
`from_avro`/`to_avro`
## What changes were proposed in this pull request?
Previously in from_avro/to_avro, we override the
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22890
This PR turn out to be a bug fix for this issue:
https://issues.apache.org/jira/browse/SPARK-26063
Back port this to branch-2.4
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/21688
Hi @pgandhi999
Thanks for the work. One minor comment here:
Currently the table header looks like this
@dbtsai @jleach4 and @aokolnychyi
---
-
To
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22966
@dbtsai Great!
I was thinking the benchmark in this PR is kind of simple, so I didn't add
it for over months..
The benchmark you mentioned should also workable for other data so
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22965#discussion_r231766852
--- Diff: sql/core/benchmarks/DataSourceReadBenchmark-results.txt ---
@@ -2,268 +2,268 @@
SQL Single Numeric Column Scan
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22965#discussion_r231765680
--- Diff: sql/core/benchmarks/DataSourceReadBenchmark-results.txt ---
@@ -2,268 +2,268 @@
SQL Single Numeric Column Scan
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22966
Done, @dongjoon-hyun PTAL.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22965
@dongjoon-hyun @yucai
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22966
@dongjoon-hyun
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22965#discussion_r231549251
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcReadBenchmark.scala ---
@@ -32,9 +32,11 @@ import org.apache.spark.sql.types
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22966
[PARK-25965][SQL] Add avro read benchmark
## What changes were proposed in this pull request?
Add read benchmark for Avro, which is missing for a period.
The benchmark is similar
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22965#discussion_r231541613
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcReadBenchmark.scala ---
@@ -266,8 +268,9 @@ object OrcReadBenchmark extends
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22965
[SPARK-25964][SQL][Minor] Revise OrcReadBenchmark/DataSourceReadBenchmark
case names and execution instructions
## What changes were proposed in this pull request?
1
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22914
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r230041592
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroEncoder.scala ---
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22914
> Returning to current page seems more code change is required. Because we
are getting all the parameters related to page from the url. Currently we are
not passing any parameter to get wh
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22864
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229952971
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,185 @@ class AvroSuite extends QueryTest with
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229976669
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,185 @@ class AvroSuite extends QueryTest with
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22914
> May be we can highlight above the table, that "Invalid page number,
falling back to first page"
Yes, that's what I mean.
No big deal but falling back to th
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22914
I prefer to just highlight the invalid output. E.g.
