[GitHub] [spark-website] zero323 commented on pull request #358: Fix remotes URLs to point to apache/spark
zero323 commented on pull request #358: URL: https://github.com/apache/spark-website/pull/358#issuecomment-939361223 Thanks @srowen! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen closed pull request #358: Fix remotes URLs to point to apache/spark
srowen closed pull request #358: URL: https://github.com/apache/spark-website/pull/358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Fix remotes URLs to point to apache/spark
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 3cff519 Fix remotes URLs to point to apache/spark 3cff519 is described below commit 3cff5195eb37664b3f6c4cd7ae664ceb45cf07aa Author: zero323 AuthorDate: Sat Oct 9 16:00:47 2021 -0500 Fix remotes URLs to point to apache/spark How to Merge a Pull Request section describes process of working with main Spark repositiory. However, `git remote` links in the How to Merge a Pull Request point to apache/spark-website. Author: zero323 Closes #358 from zero323/fix-setting-up-remotes. --- committers.md| 12 ++-- site/committers.html | 12 ++-- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/committers.md b/committers.md index ad12fa9..4bb255c 100644 --- a/committers.md +++ b/committers.md @@ -179,12 +179,12 @@ After cloning your fork of Spark you already have a remote `origin` pointing the contains at least these lines: ``` -apache g...@github.com:apache/spark-website.git (fetch) -apache g...@github.com:apache/spark-website.git (push) -apache-github g...@github.com:apache/spark-website.git (fetch) -apache-github g...@github.com:apache/spark-website.git (push) -origin g...@github.com:[your username]/spark-website.git (fetch) -origin g...@github.com:[your username]/spark-website.git (push) +apache g...@github.com:apache/spark.git (fetch) +apache g...@github.com:apache/spark.git (push) +apache-github g...@github.com:apache/spark.git (fetch) +apache-github g...@github.com:apache/spark.git (push) +origin g...@github.com:[your username]/spark.git (fetch) +origin g...@github.com:[your username]/spark.git (push) ``` For the `apache` repo, you will need to set up command-line authentication to GitHub. This may diff --git a/site/committers.html b/site/committers.html index 4e93005..93bdd5c 100644 --- a/site/committers.html +++ b/site/committers.html @@ -626,12 +626,12 @@ into the official Spark repo just by specifying your fork in the origin pointing there. So if correct, your git remote -v contains at least these lines: -apache g...@github.com:apache/spark-website.git (fetch) -apache g...@github.com:apache/spark-website.git (push) -apache-github g...@github.com:apache/spark-website.git (fetch) -apache-github g...@github.com:apache/spark-website.git (push) -origin g...@github.com:[your username]/spark-website.git (fetch) -origin g...@github.com:[your username]/spark-website.git (push) +apache g...@github.com:apache/spark.git (fetch) +apache g...@github.com:apache/spark.git (push) +apache-github g...@github.com:apache/spark.git (fetch) +apache-github g...@github.com:apache/spark.git (push) +origin g...@github.com:[your username]/spark.git (fetch) +origin g...@github.com:[your username]/spark.git (push) For the apache repo, you will need to set up command-line authentication to GitHub. This may - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] zero323 commented on pull request #358: Fix remotes URLs to point to apache/spark
zero323 commented on pull request #358: URL: https://github.com/apache/spark-website/pull/358#issuecomment-939354439 > Can you generate the HTML too? (or simply edit the HTML files too, as this is so simple) Sorry, it slipped my mind. Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #358: Fix remotes URLs to point to apache/spark
srowen commented on pull request #358: URL: https://github.com/apache/spark-website/pull/358#issuecomment-939343934 Oops, right. Can you generate the HTML too? (or simply edit the HTML files too, as this is so simple) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] zero323 opened a new pull request #358: Fix remotes URLs to point to apache/spark
zero323 opened a new pull request #358: URL: https://github.com/apache/spark-website/pull/358 How to Merge a Pull Request section describes process of working with main Spark repositiory. However, `git remote` links in the How to Merge a Pull Request point to apache/spark-website. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-36960][SQL] Pushdown filters with ANSI interval values to ORC
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ebfc6bb [SPARK-36960][SQL] Pushdown filters with ANSI interval values to ORC ebfc6bb is described below commit ebfc6bbe0e9200f87ebb52fb71d009b2d71b956d Author: Kousuke Saruta AuthorDate: Sat Oct 9 16:55:59 2021 +0300 [SPARK-36960][SQL] Pushdown filters with ANSI interval values to ORC ### What changes were proposed in this pull request? This PR proposes to push down filters with ANSI intervals to ORC. ### Why are the changes needed? After SPARK-36931 (#34184), V1 and V2 ORC datasources support ANSI intervals. So it's great to be able to push down filters with ANSI interval values for the better performance. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes #34224 from sarutak/orc-ansi-interval-pushdown. Lead-authored-by: Kousuke Saruta Co-authored-by: Kousuke Saruta Signed-off-by: Max Gekk --- .../apache/spark/sql/catalyst/dsl/package.scala| 4 +- .../sql/execution/datasources/orc/OrcFilters.scala | 10 ++- .../execution/datasources/orc/OrcFilterSuite.scala | 97 ++ 3 files changed, 108 insertions(+), 3 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala index 4a97a8d..979c280 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala @@ -18,7 +18,7 @@ package org.apache.spark.sql.catalyst import java.sql.{Date, Timestamp} -import java.time.{Instant, LocalDate} +import java.time.{Duration, Instant, LocalDate, Period} import scala.language.implicitConversions @@ -167,6 +167,8 @@ package object dsl { implicit def timestampToLiteral(t: Timestamp): Literal = Literal(t) implicit def instantToLiteral(i: Instant): Literal = Literal(i) implicit def binaryToLiteral(a: Array[Byte]): Literal = Literal(a) +implicit def periodToLiteral(p: Period): Literal = Literal(p) +implicit def durationToLiteral(d: Duration): Literal = Literal(d) implicit def symbolToUnresolvedAttribute(s: Symbol): analysis.UnresolvedAttribute = analysis.UnresolvedAttribute(s.name) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala index 5abfa4c..8e02fc3 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala @@ -17,7 +17,7 @@ package org.apache.spark.sql.execution.datasources.orc -import java.time.{Instant, LocalDate} +import java.time.{Duration, Instant, LocalDate, Period} import org.apache.hadoop.hive.common.`type`.HiveDecimal import org.apache.hadoop.hive.ql.io.sarg.{PredicateLeaf, SearchArgument} @@ -26,6 +26,7 @@ import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory.newBuilder import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable import org.apache.spark.sql.catalyst.util.DateTimeUtils.{instantToMicros, localDateToDays, toJavaDate, toJavaTimestamp} +import org.apache.spark.sql.catalyst.util.IntervalUtils import org.apache.spark.sql.errors.QueryExecutionErrors import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.sources.Filter @@ -140,7 +141,8 @@ private[sql] object OrcFilters extends OrcFiltersBase { */ def getPredicateLeafType(dataType: DataType): PredicateLeaf.Type = dataType match { case BooleanType => PredicateLeaf.Type.BOOLEAN -case ByteType | ShortType | IntegerType | LongType => PredicateLeaf.Type.LONG +case ByteType | ShortType | IntegerType | LongType | + _: AnsiIntervalType => PredicateLeaf.Type.LONG case FloatType | DoubleType => PredicateLeaf.Type.FLOAT case StringType => PredicateLeaf.Type.STRING case DateType => PredicateLeaf.Type.DATE @@ -166,6 +168,10 @@ private[sql] object OrcFilters extends OrcFiltersBase { toJavaDate(localDateToDays(value.asInstanceOf[LocalDate])) case _: TimestampType if value.isInstanceOf[Instant] => toJavaTimestamp(instantToMicros(value.asInstanceOf[Instant])) +case _: YearMonthIntervalType => + IntervalUtils.periodToMonths(value.asInstanceOf[Period]).longValue() +case _: DayTimeIntervalType => + IntervalUtils.durationToMicros(value.asInstanceOf[Duration]) case _ => value } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcF
[spark] branch master updated (4f825aa -> 7468cd7)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4f825aa [SPARK-36839][INFRA][FOLLOW-UP] Respect Hadoop version configured in scheduled GitHub Actions jobs add 7468cd7 [SPARK-36804][YARN] Support --verbose option in YARN mode No new revisions were added by this update. Summary of changes: .../org/apache/spark/deploy/yarn/ClientArguments.scala | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org