[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r248524929
##
File path: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala
##
@@ -70,26 +72,37 @@ private[ui] class EnvironmentPage(
{sparkPropertiesTable}
+
+
+
+Hadoop Properties
+
+
+
+ {hadoopPropertiesTable}
+
-
+
System Properties
-
+
Review comment:
Yes, but `collapseTablePageLoad` in `web.js` may remember actions once
expanded it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r247918057
##
File path: project/MimaExcludes.scala
##
@@ -220,6 +220,9 @@ object MimaExcludes {
// [SPARK-26139] Implement shuffle write metrics in SQL
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ShuffleDependency.this"),
+// [SPARK-26457] Show hadoop configurations in HistoryServer environment
tab
Review comment:
OK, maybe regarding hadoopConf as an optional parameters will be better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r247911946
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +445,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will not ignore configs including in Spark.
Some spark
+// conf starting with "spark.hadoop" may overwrite it.
+val hadoopProperties = hadoopConf.asScala.map(entry => (entry.getKey,
entry.getValue)).toSeq
Review comment:
`hadoopConf` in here is `org.apache.hadoop.conf.Configuration`
```
scala> import org.apache.hadoop.conf.Configuration
scala> val conf = new Configuration();
scala> conf.asScala.toSeq
res1: Seq[java.util.Map.Entry[String,String]] =
Stream(mapreduce.jobtracker.address=local, ?)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r247911946
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +445,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will not ignore configs including in Spark.
Some spark
+// conf starting with "spark.hadoop" may overwrite it.
+val hadoopProperties = hadoopConf.asScala.map(entry => (entry.getKey,
entry.getValue)).toSeq
Review comment:
`hadoopConf` in here is `org.apache.hadoop.conf.Configuration`
```suggestion
scala> import org.apache.hadoop.conf.Configuration
scala> val conf = new Configuration();
scala> conf.asScala.toSeq
res1: Seq[java.util.Map.Entry[String,String]] =
Stream(mapreduce.jobtracker.address=local, ?)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab URL: https://github.com/apache/spark/pull/23486#discussion_r247757172 ## File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ## @@ -352,6 +352,7 @@ class VersionInfo private[spark]( class ApplicationEnvironmentInfo private[spark] ( val runtime: RuntimeInfo, val sparkProperties: Seq[(String, String)], +val hadoopProperties: Seq[(String, String)], Review comment: Thank you! I will try it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r247750754
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +445,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will not ignore configs including in Spark.
Some spark
+// conf starting with "spark.hadoop" may overwrite it.
+val hadoopProperties = hadoopConf.asScala.map(entry => (entry.getKey,
entry.getValue)).toSeq
Review comment:
`Seq[(String, String)] ` will not match `Seq[Map.Entry]` here. Maybe it will
work when we use:
> val hadoopProperties = hadoopConf.asScala.toSeq.map(entry =>
(entry.getKey, entry.getValue))
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab URL: https://github.com/apache/spark/pull/23486#discussion_r247334277 ## File path: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ## @@ -352,6 +352,7 @@ class VersionInfo private[spark]( class ApplicationEnvironmentInfo private[spark] ( val runtime: RuntimeInfo, val sparkProperties: Seq[(String, String)], +val hadoopProperties: Seq[(String, String)], Review comment: Yes, you are right. Thank you! I have repaired it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r246297602
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +445,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will ignore configs including in Spark
+val hadoopProperties = hadoopConf.asScala.filter(entry =>
!conf.contains("spark.hadoop." + entry.getKey)).
Review comment:
@felixcheung Yes, I argee with you. I will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r245869725
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +443,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will ignore configs including in Spark
+val hadoopProperties = hadoopConf.asScala.filter(entry =>
!conf.contains("spark.hadoop." + entry.getKey)).
Review comment:
Yes, these properties have been shown above. So I remove it in hadoop
properties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show hadoop configurations in HistoryServer environment tab
deshanxiao commented on a change in pull request #23486: [SPARK-26457] Show
hadoop configurations in HistoryServer environment tab
URL: https://github.com/apache/spark/pull/23486#discussion_r245869725
##
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##
@@ -442,9 +443,13 @@ object SparkEnv extends Logging {
val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+// Add Hadoop properties, it will ignore configs including in Spark
+val hadoopProperties = hadoopConf.asScala.filter(entry =>
!conf.contains("spark.hadoop." + entry.getKey)).
Review comment:
Yes, these properties have shown above. So I remove it in hadoop properties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
