[spark] branch master updated (53fd83a -> ed44926)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 53fd83a [MINOR][DOCS] Fix src/dest type documentation for `to_timestamp` add ed44926 [SPARK-30627][SQL] Disable all the V2 file sources by default No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +- .../org/apache/spark/sql/connector/FileDataSourceV2FallBackSuite.scala | 3 +++ .../sql/execution/datasources/orc/OrcPartitionDiscoverySuite.scala | 2 ++ 3 files changed, 6 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] HyukjinKwon closed pull request #256: Update ml-features.html
HyukjinKwon closed pull request #256: Update ml-features.html URL: https://github.com/apache/spark-website/pull/256 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ddf8315 -> 53fd83a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ddf8315 [SPARK-28962][SQL][FOLLOW-UP] Add the parameter description for the Scala function API filter add 53fd83a [MINOR][DOCS] Fix src/dest type documentation for `to_timestamp` No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ddf8315 -> 53fd83a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ddf8315 [SPARK-28962][SQL][FOLLOW-UP] Add the parameter description for the Scala function API filter add 53fd83a [MINOR][DOCS] Fix src/dest type documentation for `to_timestamp` No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3f76bd4 -> ddf8315)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3f76bd4 [SPARK-27083][SQL][FOLLOW-UP] Rename spark.sql.subquery.reuse to spark.sql.execution.subquery.reuse.enabled add ddf8315 [SPARK-28962][SQL][FOLLOW-UP] Add the parameter description for the Scala function API filter No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/functions.scala | 15 +++ 1 file changed, 15 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4847f73 -> 3f76bd4)
This is an automated email from the ASF dual-hosted git repository. lixiao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4847f73 [SPARK-30298][SQL] Respect aliases in output partitioning of projects and aggregates add 3f76bd4 [SPARK-27083][SQL][FOLLOW-UP] Rename spark.sql.subquery.reuse to spark.sql.execution.subquery.reuse.enabled No new revisions were added by this update. Summary of changes: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3228d72 -> 4847f73)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3228d72 [SPARK-30603][SQL] Move RESERVED_PROPERTIES from SupportsNamespaces and TableCatalog to CatalogV2Util add 4847f73 [SPARK-30298][SQL] Respect aliases in output partitioning of projects and aggregates No new revisions were added by this update. Summary of changes: .../execution/AliasAwareOutputPartitioning.scala | 55 ++ .../execution/aggregate/HashAggregateExec.scala| 4 +- .../aggregate/ObjectHashAggregateExec.scala| 4 +- .../execution/aggregate/SortAggregateExec.scala| 6 +- .../sql/execution/basicPhysicalOperators.scala | 5 +- .../apache/spark/sql/execution/PlannerSuite.scala | 88 ++ .../spark/sql/sources/BucketedReadSuite.scala | 14 7 files changed, 166 insertions(+), 10 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/AliasAwareOutputPartitioning.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (976946a -> 3228d72)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 976946a [SPARK-29947][SQL][FOLLOWUP] Fix table lookup cache add 3228d72 [SPARK-30603][SQL] Move RESERVED_PROPERTIES from SupportsNamespaces and TableCatalog to CatalogV2Util No new revisions were added by this update. Summary of changes: .../sql/connector/catalog/SupportsNamespaces.java | 19 +++--- .../spark/sql/connector/catalog/TableCatalog.java | 16 +++- .../sql/connector/catalog/CatalogV2Util.scala | 29 ++ .../catalyst/analysis/ResolveSessionCatalog.scala | 6 ++--- .../apache/spark/sql/execution/command/ddl.scala | 5 ++-- .../datasources/v2/DescribeNamespaceExec.scala | 7 +++--- .../datasources/v2/DescribeTableExec.scala | 6 ++--- .../datasources/v2/V2SessionCatalog.scala | 2 +- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 12 - .../datasources/v2/V2SessionCatalogSuite.scala | 6 ++--- 10 files changed, 56 insertions(+), 52 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (843224e -> 976946a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 843224e [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature add 976946a [SPARK-29947][SQL][FOLLOWUP] Fix table lookup cache No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 8 +- .../catalyst/analysis/TableLookupCacheSuite.scala | 99 ++ 2 files changed, 103 insertions(+), 4 deletions(-) create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (afe70b3 -> 843224e)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from afe70b3 [SPARK-28794][SQL][DOC] Documentation for Create table Command add 843224e [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature No new revisions were added by this update. Summary of changes: dev/scalafmt | 5 ++--- pom.xml | 44 ++-- 2 files changed, 24 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (afe70b3 -> 843224e)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from afe70b3 [SPARK-28794][SQL][DOC] Documentation for Create table Command add 843224e [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature No new revisions were added by this update. Summary of changes: dev/scalafmt | 5 ++--- pom.xml | 44 ++-- 2 files changed, 24 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #256: Update ml-features.html
DataWanderer commented on issue #256: Update ml-features.html URL: https://github.com/apache/spark-website/pull/256#issuecomment-577820232 @HyukjinKwon Please review and submit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer removed a comment on issue #256: Update ml-features.html
DataWanderer removed a comment on issue #256: Update ml-features.html URL: https://github.com/apache/spark-website/pull/256#issuecomment-577763481 This is ostrich syndrome, simply not accepting a wasp in your repository. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3c8b360 -> afe70b3)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3c8b360 [SPARK-30620][SQL] avoid unnecessary serialization in AggregateExpression add afe70b3 [SPARK-28794][SQL][DOC] Documentation for Create table Command No new revisions were added by this update. Summary of changes: docs/sql-ref-syntax-ddl-create-table-datasource.md | 115 +++ docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 122 + docs/sql-ref-syntax-ddl-create-table-like.md | 97 docs/sql-ref-syntax-ddl-create-table.md| 12 +- 4 files changed, 345 insertions(+), 1 deletion(-) create mode 100644 docs/sql-ref-syntax-ddl-create-table-datasource.md create mode 100644 docs/sql-ref-syntax-ddl-create-table-hiveformat.md create mode 100644 docs/sql-ref-syntax-ddl-create-table-like.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3d7359a -> 3c8b360)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3d7359a [SPARK-29175][SQL][FOLLOW-UP] Rename the config name to spark.sql.maven.additionalRemoteRepositories add 3c8b360 [SPARK-30620][SQL] avoid unnecessary serialization in AggregateExpression No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-30556][SQL][2.4] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 2fc562c [SPARK-30556][SQL][2.4] Copy sparkContext.localproperties to
child thread inSubqueryExec.executionContext
2fc562c is described below
commit 2fc562cafd71ec8f438f37a28b65118906ab2ad2
Author: Ajith
AuthorDate: Thu Jan 23 09:00:01 2020 -0800
[SPARK-30556][SQL][2.4] Copy sparkContext.localproperties to child thread
inSubqueryExec.executionContext
### What changes were proposed in this pull request?
In `org.apache.spark.sql.execution.SubqueryExec#relationFuture` make a copy
of `org.apache.spark.SparkContext#localProperties` and pass it to the
sub-execution thread in
`org.apache.spark.sql.execution.SubqueryExec#executionContext`
### Why are the changes needed?
Local properties set via sparkContext are not available as TaskContext
properties when executing jobs and threadpools have idle threads which are
reused
Explanation:
When `SubqueryExec`, the relationFuture is evaluated via a separate thread.
The threads inherit the `localProperties` from `sparkContext` as they are the
child threads.
These threads are created in the `executionContext` (thread pools). Each
Thread pool has a default keepAliveSeconds of 60 seconds for idle threads.
Scenarios where the thread pool has threads which are idle and reused for a
subsequent new query, the thread local properties will not be inherited from
spark context (thread properties are inherited only on thread creation) hence
end up having old or no properties set. This will cause taskset properties to
be missing when properties are transferred by child thread via
`sparkContext.runJob/submitJob`
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Added UT
Closes #27340 from ajithme/subquerylocalprop2.
Authored-by: Ajith
Signed-off-by: Dongjoon Hyun
---
.../main/scala/org/apache/spark/util/Utils.scala | 7
.../apache/spark/sql/internal/StaticSQLConf.scala | 7
.../apache/spark/sql/execution/SQLExecution.scala | 20 +++-
.../sql/execution/basicPhysicalOperators.scala | 10 --
.../sql/internal/ExecutorSideSQLConfSuite.scala| 38 --
5 files changed, 76 insertions(+), 6 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala
b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 8f86b47..2e51614 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -2863,6 +2863,13 @@ private[spark] object Utils extends Logging {
def stringHalfWidth(str: String): Int = {
if (str == null) 0 else str.length + fullWidthRegex.findAllIn(str).size
}
+
+ /** Create a new properties object with the same values as `props` */
+ def cloneProperties(props: Properties): Properties = {
+val resultProps = new Properties()
+props.asScala.foreach(entry => resultProps.put(entry._1, entry._2))
+resultProps
+ }
}
private[util] object CallerContext extends Logging {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index d9c354b..4b5bb85 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -126,4 +126,11 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val SUBQUERY_MAX_THREAD_THRESHOLD =
+buildStaticConf("spark.sql.subquery.maxThreadThreshold")
+ .internal()
+ .doc("The maximum degree of parallelism to execute the subquery.")
+ .intConf
+ .checkValue(thres => thres > 0 && thres <= 128, "The threshold must be
in (0,128].")
+ .createWithDefault(16)
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
index 439932b..296076d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
@@ -20,9 +20,11 @@ package org.apache.spark.sql.execution
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicLong
-import org.apache.spark.SparkContext
+import scala.concurrent.{ExecutionContext, Future}
+
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.execution.ui.{SparkListenerSQLExecutionEnd,
SparkListenerSQLExecutionStart}
+import org.apache.spark.util.Utils
object SQLExecution {
@@ -129,4 +131,20 @@ object SQLExecution {
}
}
}
+
[spark] branch branch-2.4 updated (0300d4b -> c6b02cf)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 0300d4b [SPARK-30553][DOCS] fix structured-streaming java example error add c6b02cf [SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository No new revisions were added by this update. Summary of changes: .github/workflows/branch-2.4.yml | 8 pom.xml | 32 project/SparkBuild.scala | 3 +++ 3 files changed, 27 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (0300d4b -> c6b02cf)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 0300d4b [SPARK-30553][DOCS] fix structured-streaming java example error add c6b02cf [SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository No new revisions were added by this update. Summary of changes: .github/workflows/branch-2.4.yml | 8 pom.xml | 32 project/SparkBuild.scala | 3 +++ 3 files changed, 27 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #256: Update ml-features.html
DataWanderer commented on issue #256: Update ml-features.html URL: https://github.com/apache/spark-website/pull/256#issuecomment-577763481 This is ostrich syndrome, simply not accepting a wasp in your repository. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (705fc6a -> 3d7359a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 705fc6a [SPARK-30188][SQL][TESTS][FOLLOW-UP] Remove `sorted` in asserts of comparing two strings add 3d7359a [SPARK-29175][SQL][FOLLOW-UP] Rename the config name to spark.sql.maven.additionalRemoteRepositories No new revisions were added by this update. Summary of changes: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer opened a new pull request #256: Update ml-features.html
DataWanderer opened a new pull request #256: Update ml-features.html URL: https://github.com/apache/spark-website/pull/256 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #254: Update ml-features.html
DataWanderer commented on issue #254: Update ml-features.html URL: https://github.com/apache/spark-website/pull/254#issuecomment-577760185 I would not create if you answered #249. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on issue #255: Update ml-features.html
srowen commented on issue #255: Update ml-features.html URL: https://github.com/apache/spark-website/pull/255#issuecomment-577759896 That's the fourth time now. This is antagonistic and unwelcome behavior. I will have to block you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen closed pull request #255: Update ml-features.html
srowen closed pull request #255: Update ml-features.html URL: https://github.com/apache/spark-website/pull/255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on issue #249: Update ml-features.html
srowen commented on issue #249: Update ml-features.html URL: https://github.com/apache/spark-website/pull/249#issuecomment-577759700 I answered you above. This is not the place to make the change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #255: Update ml-features.html
DataWanderer commented on issue #255: Update ml-features.html URL: https://github.com/apache/spark-website/pull/255#issuecomment-577758987 Hi @HyukjinKwon Kindly review , thank you in advance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer opened a new pull request #255: Update ml-features.html
DataWanderer opened a new pull request #255: Update ml-features.html URL: https://github.com/apache/spark-website/pull/255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer edited a comment on issue #250: Update ml-features.html
DataWanderer edited a comment on issue #250: Update ml-features.html URL: https://github.com/apache/spark-website/pull/250#issuecomment-577757884 @srowen can you answer #249 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #250: Update ml-features.html
DataWanderer commented on issue #250: Update ml-features.html URL: https://github.com/apache/spark-website/pull/250#issuecomment-577757884 can you answer #249 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer commented on issue #249: Update ml-features.html
DataWanderer commented on issue #249: Update ml-features.html URL: https://github.com/apache/spark-website/pull/249#issuecomment-577755839 @srowen so you are saying this is not a wasp in your repository? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] DataWanderer opened a new pull request #254: Update ml-features.html
DataWanderer opened a new pull request #254: Update ml-features.html URL: https://github.com/apache/spark-website/pull/254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen closed pull request #254: Update ml-features.html
srowen closed pull request #254: Update ml-features.html URL: https://github.com/apache/spark-website/pull/254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on issue #254: Update ml-features.html
srowen commented on issue #254: Update ml-features.html URL: https://github.com/apache/spark-website/pull/254#issuecomment-577755307 Third time you've opened the same change: #249 #250 Same answer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Annotating link to Spark PRs page with
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new b8ff0d6 Annotating link to Spark PRs page with b8ff0d6 is described below commit b8ff0d64469270e7af2e809f23fddd71f7e4ea34 Author: Enrico Minack AuthorDate: Thu Jan 23 09:10:56 2020 -0600 Annotating link to Spark PRs page with The link can not be clicked otherwise. Author: Enrico Minack Closes #253 from EnricoMi/patch-1. --- contributing.md| 3 ++- site/contributing.html | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/contributing.md b/contributing.md index ced3d9c..9cde6ce 100644 --- a/contributing.md +++ b/contributing.md @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a +convenient way to view and filter open PRs. Contributing Documentation Changes diff --git a/site/contributing.html b/site/contributing.html index a8cf2fe..b29d381 100644 --- a/site/contributing.html +++ b/site/contributing.html @@ -241,7 +241,8 @@ Anyone can view and comment on active changes here. Reviewing others changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a +convenient way to view and filter open PRs. Contributing Documentation Changes - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen closed pull request #253: Annotating link to Spark PRs page with
srowen closed pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on a change in pull request #253: Annotating link to Spark PRs page with
srowen commented on a change in pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253#discussion_r370172877 ## File path: contributing.md ## @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a Review comment: It can use markdown link syntax too like `[..](...)` but it doesn't matter. Seems fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dbed4c7 -> 705fc6a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dbed4c7 [SPARK-30605][SQL] move defaultNamespace from SupportsNamespace to CatalogPlugin add 705fc6a [SPARK-30188][SQL][TESTS][FOLLOW-UP] Remove `sorted` in asserts of comparing two strings No new revisions were added by this update. Summary of changes: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ffd435b -> dbed4c7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ffd435b [SPARK-27871][SQL][FOLLOW-UP] Remove the conf spark.sql.optimizer.reassignLambdaVariableID.enabled add dbed4c7 [SPARK-30605][SQL] move defaultNamespace from SupportsNamespace to CatalogPlugin No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/CatalogPlugin.java | 14 .../catalog/DelegatingCatalogExtension.java| 10 - .../sql/connector/catalog/SupportsNamespaces.java | 14 .../sql/connector/catalog/CatalogManager.scala | 7 +- .../connector/catalog/CatalogManagerSuite.scala| 25 ++ 5 files changed, 22 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0bf447 -> ffd435b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0bf447 [SPARK-30575][DOCS][FOLLOWUP] Fix typos in documents add ffd435b [SPARK-27871][SQL][FOLLOW-UP] Remove the conf spark.sql.optimizer.reassignLambdaVariableID.enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/catalyst/optimizer/objects.scala | 2 -- .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 7 --- .../scala/org/apache/spark/sql/DatasetOptimizationSuite.scala| 9 - 3 files changed, 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] EnricoMi commented on a change in pull request #253: Annotating link to Spark PRs page with
EnricoMi commented on a change in pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253#discussion_r370013998 ## File path: contributing.md ## @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on a change in pull request #253: Annotating link to Spark PRs page with
HyukjinKwon commented on a change in pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253#discussion_r370012102 ## File path: contributing.md ## @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a Review comment: Yeah, it happens due to different versions. You can manually exclude unrelated changes except this line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] EnricoMi commented on a change in pull request #253: Annotating link to Spark PRs page with
EnricoMi commented on a change in pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253#discussion_r370008130 ## File path: contributing.md ## @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a Review comment: I did, I run jekyll 4.0.0 and it generated a lot of unrelated changes. I will try again with <4.0.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on a change in pull request #253: Annotating link to Spark PRs page with
HyukjinKwon commented on a change in pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253#discussion_r370007200 ## File path: contributing.md ## @@ -48,7 +48,8 @@ Anyone can view and comment on active changes here. Reviewing others' changes is a good way to learn how the change process works and gain exposure to activity in various parts of the code. You can help by reviewing the changes and asking questions or pointing out issues -- as simple as typos or small issues of style. -See also https://spark-prs.appspot.com/ for a convenient way to view and filter open PRs. +See also https://spark-prs.appspot.com/;>https://spark-prs.appspot.com/ for a Review comment: @EnricoMi, can you generate HTML too accordingly? See https://github.com/apache/spark-website#generating-the-website-html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f35f352 -> d0bf447)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f35f352 [SPARK-30543][ML][PYSPARK][R] RandomForest add Param bootstrap to control sampling method add d0bf447 [SPARK-30575][DOCS][FOLLOWUP] Fix typos in documents No new revisions were added by this update. Summary of changes: docs/sql-ref-syntax-qry-select-groupby.md | 2 +- docs/sql-ref-syntax-qry-select-having.md | 4 ++-- docs/sql-ref-syntax-qry-select-where.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2330a56 -> f35f352)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2330a56 [SPARK-30607][SQL][PYSPARK][SPARKR] Add overlay wrappers for SparkR and PySpark add f35f352 [SPARK-30543][ML][PYSPARK][R] RandomForest add Param bootstrap to control sampling method No new revisions were added by this update. Summary of changes: R/pkg/R/mllib_tree.R | 9 ++--- R/pkg/tests/fulltests/test_mllib_tree.R | 2 +- .../ml/classification/DecisionTreeClassifier.scala | 1 + .../ml/classification/RandomForestClassifier.scala | 7 ++- .../ml/r/RandomForestClassificationWrapper.scala | 4 +++- .../spark/ml/r/RandomForestRegressionWrapper.scala | 4 +++- .../spark/ml/regression/AFTSurvivalRegression.scala | 2 +- .../spark/ml/regression/DecisionTreeRegressor.scala | 2 +- .../org/apache/spark/ml/regression/FMRegressor.scala | 2 +- .../apache/spark/ml/regression/GBTRegressor.scala| 1 - .../spark/ml/regression/RandomForestRegressor.scala | 8 ++-- .../org/apache/spark/ml/tree/impl/BaggedPoint.scala | 18 +++--- .../spark/ml/tree/impl/GradientBoostedTrees.scala| 5 +++-- .../org/apache/spark/ml/tree/impl/RandomForest.scala | 4 +--- .../scala/org/apache/spark/ml/tree/treeParams.scala | 14 ++ .../org/apache/spark/mllib/tree/RandomForest.scala | 8 +++- .../spark/mllib/tree/configuration/Strategy.scala| 3 ++- .../classification/RandomForestClassifierSuite.scala | 5 - .../ml/regression/RandomForestRegressorSuite.scala | 2 ++ python/pyspark/ml/classification.py | 20 +++- python/pyspark/ml/regression.py | 20 +++- python/pyspark/ml/tree.py| 12 +++- 22 files changed, 114 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2330a56 -> f35f352)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2330a56 [SPARK-30607][SQL][PYSPARK][SPARKR] Add overlay wrappers for SparkR and PySpark add f35f352 [SPARK-30543][ML][PYSPARK][R] RandomForest add Param bootstrap to control sampling method No new revisions were added by this update. Summary of changes: R/pkg/R/mllib_tree.R | 9 ++--- R/pkg/tests/fulltests/test_mllib_tree.R | 2 +- .../ml/classification/DecisionTreeClassifier.scala | 1 + .../ml/classification/RandomForestClassifier.scala | 7 ++- .../ml/r/RandomForestClassificationWrapper.scala | 4 +++- .../spark/ml/r/RandomForestRegressionWrapper.scala | 4 +++- .../spark/ml/regression/AFTSurvivalRegression.scala | 2 +- .../spark/ml/regression/DecisionTreeRegressor.scala | 2 +- .../org/apache/spark/ml/regression/FMRegressor.scala | 2 +- .../apache/spark/ml/regression/GBTRegressor.scala| 1 - .../spark/ml/regression/RandomForestRegressor.scala | 8 ++-- .../org/apache/spark/ml/tree/impl/BaggedPoint.scala | 18 +++--- .../spark/ml/tree/impl/GradientBoostedTrees.scala| 5 +++-- .../org/apache/spark/ml/tree/impl/RandomForest.scala | 4 +--- .../scala/org/apache/spark/ml/tree/treeParams.scala | 14 ++ .../org/apache/spark/mllib/tree/RandomForest.scala | 8 +++- .../spark/mllib/tree/configuration/Strategy.scala| 3 ++- .../classification/RandomForestClassifierSuite.scala | 5 - .../ml/regression/RandomForestRegressorSuite.scala | 2 ++ python/pyspark/ml/classification.py | 20 +++- python/pyspark/ml/regression.py | 20 +++- python/pyspark/ml/tree.py| 12 +++- 22 files changed, 114 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] EnricoMi opened a new pull request #253: Annotating link to Spark PRs page with
EnricoMi opened a new pull request #253: Annotating link to Spark PRs page with URL: https://github.com/apache/spark-website/pull/253 The link can not be clicked otherwise. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
