[GitHub] [spark] SparkQA commented on pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
SparkQA commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-975217562 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49972/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile
SparkQA commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-975217339 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49973/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
SparkQA commented on pull request #34060: URL: https://github.com/apache/spark/pull/34060#issuecomment-975216033 **[Test build #145505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145505/testReport)** for PR 34060 at commit [`a2dd853`](https://github.com/apache/spark/commit/a2dd853d3ab8d17cc26d50b183d8363f00ad8bbc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
SparkQA commented on pull request #34513: URL: https://github.com/apache/spark/pull/34513#issuecomment-975215676 **[Test build #145504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145504/testReport)** for PR 34513 at commit [`836903a`](https://github.com/apache/spark/commit/836903af42aae90e333ee04af7ae1170bcfbce34). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34680: [SPARK-37421][PYTHON] Inline type hints for python/pyspark/mllib/evaluation.py
SparkQA commented on pull request #34680: URL: https://github.com/apache/spark/pull/34680#issuecomment-975215469 **[Test build #145503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145503/testReport)** for PR 34680 at commit [`e686682`](https://github.com/apache/spark/commit/e68668213b80063a45a60af2174244d55068f222). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins removed a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975214010 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49968/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
AmplabJenkins removed a comment on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975214012 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145492/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
AmplabJenkins removed a comment on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975214009 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145497/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
AmplabJenkins commented on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975214009 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145497/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975214010 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49968/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
AmplabJenkins commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975214012 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145492/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
cloud-fan commented on a change in pull request #34666:
URL: https://github.com/apache/spark/pull/34666#discussion_r754016866
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTblPropertiesSuite.scala
##
@@ -56,7 +44,7 @@ trait ShowTblPropertiesSuiteBase extends
command.ShowTblPropertiesSuiteBase
}
}
- test("SHOW TBLPROPERTIES FOR TEMPORARY IEW") {
+ testV1("SHOW TBLPROPERTIES FOR TEMPORARY VIEW") {
Review comment:
are you saying, after this PR, by default `SHOW TBLPROPERTIES` can't
work with temp view? That's a terrible breaking change and we should fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
SparkQA commented on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975213361 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49969/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
cloud-fan commented on a change in pull request #34666:
URL: https://github.com/apache/spark/pull/34666#discussion_r754016866
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTblPropertiesSuite.scala
##
@@ -56,7 +44,7 @@ trait ShowTblPropertiesSuiteBase extends
command.ShowTblPropertiesSuiteBase
}
}
- test("SHOW TBLPROPERTIES FOR TEMPORARY IEW") {
+ testV1("SHOW TBLPROPERTIES FOR TEMPORARY VIEW") {
Review comment:
are you saying, by default `SHOW TBLPROPERTIES` can't work with temp
view? That's a terrible breaking change and we should fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
cloud-fan commented on a change in pull request #34666:
URL: https://github.com/apache/spark/pull/34666#discussion_r754015958
##
File path: sql/core/src/test/resources/sql-tests/inputs/show-tblproperties.sql
##
@@ -1,26 +0,0 @@
--- create a table with properties
-CREATE TABLE tbl (a INT, b STRING, c INT) USING parquet
-TBLPROPERTIES('p1'='v1', 'p2'='v2');
-
-SHOW TBLPROPERTIES tbl;
-SHOW TBLPROPERTIES tbl("p1");
-SHOW TBLPROPERTIES tbl("p3");
-
-DROP TABLE tbl;
-
--- create a view with properties
-CREATE VIEW view TBLPROPERTIES('p1'='v1', 'p2'='v2') AS SELECT 1 AS c1;
Review comment:
does the `ShowTblPropertiesSuiteBase` cover view?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
SparkQA removed a comment on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975022788 **[Test build #145492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145492/testReport)** for PR 34532 at commit [`9436901`](https://github.com/apache/spark/commit/94369012c9da159d4aef2f68fa965a4c9d602e3d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
SparkQA commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975210052 **[Test build #145492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145492/testReport)** for PR 34532 at commit [`9436901`](https://github.com/apache/spark/commit/94369012c9da159d4aef2f68fa965a4c9d602e3d). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):` * `class ProbabilisticClassificationModel(` * `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):` * `class _JavaProbabilisticClassificationModel(` * `class _LinearSVCParams(` * `class LinearSVCModel(` * `class _LogisticRegressionParams(` * `class LogisticRegression(` * `class LogisticRegressionModel(` * `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):` * `class BinaryLogisticRegressionTrainingSummary(` * `class DecisionTreeClassifier(` * `class DecisionTreeClassificationModel(` * `class RandomForestClassifier(` * `class RandomForestClassificationModel(` * `class RandomForestClassificationTrainingSummary(` * `class BinaryRandomForestClassificationTrainingSummary(` * `class GBTClassifier(` * `class GBTClassificationModel(` * `class NaiveBayes(` * `class NaiveBayesModel(` * `class _MultilayerPerceptronParams(` * `class MultilayerPerceptronClassifier(` * `class MultilayerPerceptronClassificationModel(` * `class MultilayerPerceptronClassificationTrainingSummary(` * `class FMClassifier(` * `class FMClassificationModel(` * `class _GaussianMixtureParams(` * `class GaussianMixtureModel(` * `class _KMeansParams(` * `class KMeansModel(` * `class _BisectingKMeansParams(` * `class BisectingKMeansModel(` * `class PowerIterationClustering(` * `class BinaryClassificationEvaluator(` * `class RegressionEvaluator(` * `class MulticlassClassificationEvaluator(` * `class MultilabelClassificationEvaluator(` * `class ClusteringEvaluator(` * `class RankingEvaluator(` * `class Binarizer(` * `class BucketedRandomProjectionLSH(` * `class BucketedRandomProjectionLSHModel(` * `class Bucketizer(` * `class ElementwiseProduct(` * `class FeatureHasher(` * `class HashingTF(` * `class _OneHotEncoderParams(` * `class PolynomialExpansion(` * `class QuantileDiscretizer(` * `class _StringIndexerParams(` * `class StopWordsRemover(` * `class VectorAssembler(` * `class VectorSizeHint(` * `class VarianceThresholdSelector(` * `class VarianceThresholdSelectorModel(` * `class UnivariateFeatureSelector(` * `class UnivariateFeatureSelectorModel(` * `class _LinearRegressionParams(` * `class LinearRegressionModel(` * `class IsotonicRegression(` * `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):` * `class DecisionTreeRegressor(` * `class RandomForestRegressor(` * `class _AFTSurvivalRegressionParams(` * `class AFTSurvivalRegression(` * `class AFTSurvivalRegressionModel(` * `class _GeneralizedLinearRegressionParams(` * `class GeneralizedLinearRegression(` * `class GeneralizedLinearRegressionModel(` * `class _FactorizationMachinesParams(` * `class FMRegressionModel(` * `class CrossValidator(` * `class TrainValidationSplit(` * `+ \"class name ` * `class PandasAPIOnSparkAdviceWarning(Warning):` * `class ArrowStreamUDFSerializer(ArrowStreamSerializer):` * `class DayTimeIntervalType(AtomicType):` * `class DayTimeIntervalTypeConverter(object):` * `case class ExpressionStats(expr: Expression)(var useCount: Int)` * `case class PythonMapInArrow(` * `case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)` * `trait MapInBatchExec extends UnaryExecNode ` * `case class PythonMapInArrowExec(` * ` // When this is enabled, this class does additional lookup on write operations (put/delete) to` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
SparkQA commented on pull request #34666: URL: https://github.com/apache/spark/pull/34666#issuecomment-975209500 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49971/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975208219 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49968/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
SparkQA commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-975208165 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49970/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34680: [SPARK-37421][PYTHON] Inline type hints for python/pyspark/mllib/evaluation.py
dchvn commented on a change in pull request #34680:
URL: https://github.com/apache/spark/pull/34680#discussion_r754003277
##
File path: python/pyspark/mllib/evaluation.py
##
@@ -74,29 +80,31 @@ def __init__(self, scoreAndLabels):
if numCol == 3:
schema.add("weight", DoubleType(), False)
df = sql_ctx.createDataFrame(scoreAndLabels, schema=schema)
-java_class =
sc._jvm.org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
+java_class = (
+
sc._jvm.org.apache.spark.mllib.evaluation.BinaryClassificationMetrics # type:
ignore[attr-defined]
+)
java_model = java_class(df._jdf)
super(BinaryClassificationMetrics, self).__init__(java_model)
-@property
+@property # type: ignore[misc]
Review comment:
I met `error: Decorated property not supported [misc]` when checking
mypy for every `@property`. As mypy's issue
[#1362](https://github.com/python/mypy/issues/1362), I think we should ignore
this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34680: [SPARK-37421][PYTHON] Inline type hints for python/pyspark/mllib/evaluation.py
dchvn commented on pull request #34680: URL: https://github.com/apache/spark/pull/34680#issuecomment-975196185 cc @zero323, @HyukjinKwon, @ueshin FYI. Many thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
SparkQA removed a comment on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975181888 **[Test build #145497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145497/testReport)** for PR 34678 at commit [`609831c`](https://github.com/apache/spark/commit/609831c5753f46cefb67f7e1f680ff3621661178). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
dchvn commented on a change in pull request #34513: URL: https://github.com/apache/spark/pull/34513#discussion_r754001496 ## File path: python/pyspark/mllib/stat/_statistics.py ## @@ -16,51 +16,57 @@ # import sys +from typing import overload, List, Optional, Union +from typing_extensions import Literal Review comment: thanks! updated. I also move ```python DistName = Literal["norm"] ``` to ```_typing.pyi``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
SparkQA commented on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975195667 **[Test build #145497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145497/testReport)** for PR 34678 at commit [`609831c`](https://github.com/apache/spark/commit/609831c5753f46cefb67f7e1f680ff3621661178). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34680: [SPARK-37421][PYTHON] Inline type hints for python/pyspark/mllib/evaluation.py
dchvn commented on a change in pull request #34680:
URL: https://github.com/apache/spark/pull/34680#discussion_r754000217
##
File path: python/pyspark/mllib/evaluation.py
##
@@ -74,29 +80,31 @@ def __init__(self, scoreAndLabels):
if numCol == 3:
schema.add("weight", DoubleType(), False)
df = sql_ctx.createDataFrame(scoreAndLabels, schema=schema)
-java_class =
sc._jvm.org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
+java_class = (
+
sc._jvm.org.apache.spark.mllib.evaluation.BinaryClassificationMetrics # type:
ignore[attr-defined]
+)
java_model = java_class(df._jdf)
super(BinaryClassificationMetrics, self).__init__(java_model)
-@property
+@property # type: ignore[misc]
Review comment:
I met `Decorated property not supported [misc]` for checking mypy. As
mypy's issue [#1362](https://github.com/python/mypy/issues/1362), I think we
should ignore this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
dchvn commented on a change in pull request #34513: URL: https://github.com/apache/spark/pull/34513#discussion_r753999742 ## File path: python/pyspark/mllib/linalg/__init__.pyi ## @@ -68,6 +68,7 @@ class Vector: __UDT__: VectorUDT def toArray(self) -> ndarray: ... def asML(self) -> newlinalg.Vector: ... +def __len__(self) -> int: ... Review comment: thanks! updated with ```ignore``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn opened a new pull request #34680: [SPARK-37421][PYTHON] Inline type hints for python/pyspark/mllib/evaluation.py
dchvn opened a new pull request #34680: URL: https://github.com/apache/spark/pull/34680 ### What changes were proposed in this pull request? Inline type hints for evaluation.py in python/pyspark/mllib/ ### Why are the changes needed? We can take advantage of static type checking within the functions by inlining the type hints. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
SparkQA commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-975190436 **[Test build #145502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145502/testReport)** for PR 34677 at commit [`091bd96`](https://github.com/apache/spark/commit/091bd96da2738a50e4a6a5f02af91a3fc52c8701). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
cloud-fan commented on a change in pull request #34668:
URL: https://github.com/apache/spark/pull/34668#discussion_r753996710
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -73,6 +73,21 @@ grammar SqlBase;
return false;
}
}
+
+ /**
+ * This method will be called when the character stream ends and try to find
out the
+ * unclosed bracketed comment.
+ * If the next character is -1, it means the end of the entire character
stream match,
+ * and we throw exception to prevent executing the query.
+ */
+ public void end() {
+int nextChar = _input.LA(1);
+if (nextChar == -1) {
+ int pos = _input.index();
+ String str = _input.getText(new Interval(0, pos));
+ throw new RuntimeException("Unclosed bracketed comment: " + str + ",
position: " + pos);
Review comment:
And we can throw ParserException? and put the error in
`QueryParsingErrors`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
cloud-fan commented on a change in pull request #34668:
URL: https://github.com/apache/spark/pull/34668#discussion_r753996055
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -73,6 +73,21 @@ grammar SqlBase;
return false;
}
}
+
+ /**
+ * This method will be called when the character stream ends and try to find
out the
+ * unclosed bracketed comment.
+ * If the next character is -1, it means the end of the entire character
stream match,
+ * and we throw exception to prevent executing the query.
+ */
+ public void end() {
+int nextChar = _input.LA(1);
+if (nextChar == -1) {
+ int pos = _input.index();
+ String str = _input.getText(new Interval(0, pos));
+ throw new RuntimeException("Unclosed bracketed comment: " + str + ",
position: " + pos);
Review comment:
can we get the start position instead of the end position?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
cloud-fan commented on a change in pull request #34668:
URL: https://github.com/apache/spark/pull/34668#discussion_r753995720
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -73,6 +73,21 @@ grammar SqlBase;
return false;
}
}
+
+ /**
+ * This method will be called when the character stream ends and try to find
out the
+ * unclosed bracketed comment.
+ * If the next character is -1, it means the end of the entire character
stream match,
+ * and we throw exception to prevent executing the query.
+ */
+ public void end() {
+int nextChar = _input.LA(1);
+if (nextChar == -1) {
+ int pos = _input.index();
+ String str = _input.getText(new Interval(0, pos));
+ throw new RuntimeException("Unclosed bracketed comment: " + str + ",
position: " + pos);
Review comment:
since the sql string can be multi-line, we can make the error message
more readable
```
Unclosed bracketed comment started at position $pos:
$str
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753995410
##
File path: python/pyspark/mllib/stat/_statistics.py
##
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector),
method).toArray()
else:
-return callMLlibFunc("corr", x.map(float), y.map(float), method)
+return callMLlibFunc(
+"corr", x.map(float), y.map(float), method # type:
ignore[arg-type]
Review comment:
```python
Argument 1 to "map" of "RDD" has incompatible type "Type[float]"; expected
"Callable[[Vector], Any]" [arg-type]
```
I got that error if it is not ignored, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
SparkQA commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-975185264 **[Test build #145501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145501/testReport)** for PR 34677 at commit [`bace4c0`](https://github.com/apache/spark/commit/bace4c01930b903d53616ee6bd65b4b762941cc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34679: [SPARK-37437][BUILD] remove unused hive profile
SparkQA commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-975185148 **[Test build #145500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145500/testReport)** for PR 34679 at commit [`f78af65`](https://github.com/apache/spark/commit/f78af6584fa1f899dbab6e3abd2db07dcb22d022). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #34679: [SPARK-37437][BUILD] remove unused hive profile
AngersZh opened a new pull request #34679: URL: https://github.com/apache/spark/pull/34679 ### What changes were proposed in this pull request? Since we only support hive-2.3, we should remove the unused profile. ### Why are the changes needed? remove unused profile ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not need -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975183385 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49967/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975183385 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49967/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975183367 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49967/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
SparkQA commented on pull request #34666: URL: https://github.com/apache/spark/pull/34666#issuecomment-975181998 **[Test build #145499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145499/testReport)** for PR 34666 at commit [`6e8ac96`](https://github.com/apache/spark/commit/6e8ac96b747512d9974c541af75beda09c8506ec). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
SparkQA commented on pull request #34677: URL: https://github.com/apache/spark/pull/34677#issuecomment-975181931 **[Test build #145498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145498/testReport)** for PR 34677 at commit [`a8c3eca`](https://github.com/apache/spark/commit/a8c3eca0e29c69d893a4ed8842f13ddfe7f894fb). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
SparkQA commented on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975181888 **[Test build #145497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145497/testReport)** for PR 34678 at commit [`609831c`](https://github.com/apache/spark/commit/609831c5753f46cefb67f7e1f680ff3621661178). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975180504 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145490/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins removed a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975180497 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145494/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975180504 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145490/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975180497 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145494/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA removed a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975049090 **[Test build #145494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145494/testReport)** for PR 34676 at commit [`b8e917f`](https://github.com/apache/spark/commit/b8e917fe4d73cf6c8a78fb90764e2a1ebdb250a3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975178783 **[Test build #145494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145494/testReport)** for PR 34676 at commit [`b8e917f`](https://github.com/apache/spark/commit/b8e917fe4d73cf6c8a78fb90764e2a1ebdb250a3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975178405 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49968/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
dchvn commented on pull request #34678: URL: https://github.com/apache/spark/pull/34678#issuecomment-975177529 cc @zero323 Please take a look when you have time! Thanks 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn opened a new pull request #34678: [SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
dchvn opened a new pull request #34678: URL: https://github.com/apache/spark/pull/34678 ### What changes were proposed in this pull request? Inline type hints for python/pyspark/ml/functions.py ### Why are the changes needed? We can take advantage of static type checking within the functions by inlining the type hints. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
beliefer commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975176620 ping @cloud-fan @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
SparkQA removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975002132 **[Test build #145490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145490/testReport)** for PR 34668 at commit [`75c6e43`](https://github.com/apache/spark/commit/75c6e439483bb2692ebb4093dc9a60fc9f78c728). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975174898 **[Test build #145490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145490/testReport)** for PR 34668 at commit [`75c6e43`](https://github.com/apache/spark/commit/75c6e439483bb2692ebb4093dc9a60fc9f78c728). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #34677: [WIP][SPARK-37436][PYTHON] Uses Python's standard string formatter for SQL API in pandas API on Spark
HyukjinKwon opened a new pull request #34677: URL: https://github.com/apache/spark/pull/34677 ### What changes were proposed in this pull request? This PR proposes to use [Python's standard string formatter](https://docs.python.org/3/library/string.html#custom-string-formatting) instead of hacky custom SQL parser for SQL API in pandas API on Spark ### Why are the changes needed? TBD ### Does this PR introduce _any_ user-facing change? TBD ### How was this patch tested? Doctests were added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
Peng-Lei commented on a change in pull request #34666:
URL: https://github.com/apache/spark/pull/34666#discussion_r753982833
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowTblPropertiesSuiteBase.scala
##
@@ -87,4 +88,25 @@ trait ShowTblPropertiesSuiteBase extends QueryTest with
DDLCommandTestUtils {
assert(res.head.getString(1).contains(s"does not have property:
$nonExistingKey"))
}
}
+
+ test("KEEP THE LEGACY OUTPUT SCHEMA") {
+Seq(true, false).foreach { keepLegacySchema =>
+ withSQLConf(SQLConf.LEGACY_KEEP_COMMAND_OUTPUT_SCHEMA.key ->
keepLegacySchema.toString) {
+withNamespaceAndTable("ns1", "tbl") { tbl =>
+ spark.sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing
" +
+s"TBLPROPERTIES ('user'='spark', 'status'='new')")
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34666: [SPARK-37192][SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
Peng-Lei commented on a change in pull request #34666:
URL: https://github.com/apache/spark/pull/34666#discussion_r753982634
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTblPropertiesSuite.scala
##
@@ -56,7 +44,7 @@ trait ShowTblPropertiesSuiteBase extends
command.ShowTblPropertiesSuiteBase
}
}
- test("SHOW TBLPROPERTIES FOR TEMPORARY IEW") {
+ testV1("SHOW TBLPROPERTIES FOR TEMPORARY IEW") {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34314: [SPARK-36231][PYTHON] Support arithmetic operations of decimal(nan) series
Yikun commented on pull request #34314: URL: https://github.com/apache/spark/pull/34314#issuecomment-975166197 > We should probably have to bump up .. ideally we should test all the combinatins just like other python projects .. but we can't do this due to the resource problem in GA. @HyukjinKwon OK, thanks! That means we soulld test it after v0.23.2. I will address soon. : ) > Testing on 1.1.x or 1.2.x should be good enough for the fix itself. OK, thanks for reminder. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34314: [SPARK-36231][PYTHON] Support arithmetic operations of decimal(nan) series
HyukjinKwon commented on pull request #34314: URL: https://github.com/apache/spark/pull/34314#issuecomment-975165011 Testing on 1.1.x or 1.2.x should be good enough for the fix itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34314: [SPARK-36231][PYTHON] Support arithmetic operations of decimal(nan) series
HyukjinKwon commented on pull request #34314: URL: https://github.com/apache/spark/pull/34314#issuecomment-975164734 It's actually documented here; https://github.com/apache/spark/blob/master/python/setup.py#L115. We should probably have to bump up .. ideally we should test all the combinatins just like other python projects .. but we can't do this due to the resource problem in GA 😢 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34314: [SPARK-36231][PYTHON] Support arithmetic operations of decimal(nan) series
Yikun commented on pull request #34314: URL: https://github.com/apache/spark/pull/34314#issuecomment-975160487 @HyukjinKwon Thanks for your help, and one more question, what's the mainly version of pandas should be tested and supported? Should we announce it in somewhere, and then add the test to install specific pandas version in CI to do an extra check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
HyukjinKwon commented on a change in pull request #34676: URL: https://github.com/apache/spark/pull/34676#discussion_r753967179 ## File path: pom.xml ## @@ -3580,6 +3580,18 @@ + + mac-on-apple-silicon + + org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest Review comment: WDYT @dongjoon-hyun ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34670: [SPARK-37388][SQL] Fix NPE in WidthBucket in WholeStageCodegenExec
AmplabJenkins removed a comment on pull request #34670: URL: https://github.com/apache/spark/pull/34670#issuecomment-975153964 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145487/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34670: [SPARK-37388][SQL] Fix NPE in WidthBucket in WholeStageCodegenExec
AmplabJenkins commented on pull request #34670: URL: https://github.com/apache/spark/pull/34670#issuecomment-975153964 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145487/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34670: [SPARK-37388][SQL] Fix NPE in WidthBucket in WholeStageCodegenExec
SparkQA removed a comment on pull request #34670: URL: https://github.com/apache/spark/pull/34670#issuecomment-974961496 **[Test build #145487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145487/testReport)** for PR 34670 at commit [`0cba30d`](https://github.com/apache/spark/commit/0cba30d995a5056cf0f3762fa0d5ec88d2282e05). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34670: [SPARK-37388][SQL] Fix NPE in WidthBucket in WholeStageCodegenExec
SparkQA commented on pull request #34670: URL: https://github.com/apache/spark/pull/34670#issuecomment-975152621 **[Test build #145487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145487/testReport)** for PR 34670 at commit [`0cba30d`](https://github.com/apache/spark/commit/0cba30d995a5056cf0f3762fa0d5ec88d2282e05). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975151828 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49967/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975150371 **[Test build #145496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145496/testReport)** for PR 34676 at commit [`f87a633`](https://github.com/apache/spark/commit/f87a633e17ac96d46ca432155faf710df4b63460). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
AmplabJenkins removed a comment on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975149860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
LuciferYang commented on a change in pull request #34676: URL: https://github.com/apache/spark/pull/34676#discussion_r753959557 ## File path: pom.xml ## @@ -3580,6 +3580,18 @@ + + mac-on-apple-silicon + + org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest Review comment: @HyukjinKwon 7a17868 and f87a633 change to re-write `test.default.exclude.tags`, in this way, we can still use `test.exclude.tags` in other scenarios. Can this ease your concerns? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins removed a comment on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975149858 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49966/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
AmplabJenkins commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975149858 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49966/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
AmplabJenkins commented on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975149860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
LuciferYang commented on a change in pull request #34676: URL: https://github.com/apache/spark/pull/34676#discussion_r753959557 ## File path: pom.xml ## @@ -3580,6 +3580,18 @@ + + mac-on-apple-silicon + + org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest Review comment: @HyukjinKwon 7a17868 change to re-write `test.default.exclude.tags`, in this way, we can still use `test.exclude.tags` in other scenarios. Can this ease your concerns? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
SparkQA removed a comment on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975023113 **[Test build #145493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145493/testReport)** for PR 33628 at commit [`1eaad94`](https://github.com/apache/spark/commit/1eaad948fd77e442b5cc8a3c8df02d9aa98025e3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
SparkQA commented on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975148251 **[Test build #145493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145493/testReport)** for PR 33628 at commit [`1eaad94`](https://github.com/apache/spark/commit/1eaad948fd77e442b5cc8a3c8df02d9aa98025e3). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkConf(object):` * `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):` * `class ProbabilisticClassificationModel(` * `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):` * `class _JavaProbabilisticClassificationModel(` * `class _LinearSVCParams(` * `class LinearSVCModel(` * `class _LogisticRegressionParams(` * `class LogisticRegression(` * `class LogisticRegressionModel(` * `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):` * `class BinaryLogisticRegressionTrainingSummary(` * `class DecisionTreeClassifier(` * `class DecisionTreeClassificationModel(` * `class RandomForestClassifier(` * `class RandomForestClassificationModel(` * `class RandomForestClassificationTrainingSummary(` * `class BinaryRandomForestClassificationTrainingSummary(` * `class GBTClassifier(` * `class GBTClassificationModel(` * `class NaiveBayes(` * `class NaiveBayesModel(` * `class _MultilayerPerceptronParams(` * `class MultilayerPerceptronClassifier(` * `class MultilayerPerceptronClassificationModel(` * `class MultilayerPerceptronClassificationTrainingSummary(` * `class FMClassifier(` * `class FMClassificationModel(` * `class _GaussianMixtureParams(` * `class GaussianMixtureModel(` * `class _KMeansParams(` * `class KMeansModel(` * `class _BisectingKMeansParams(` * `class BisectingKMeansModel(` * `class PowerIterationClustering(` * `class BinaryClassificationEvaluator(` * `class RegressionEvaluator(` * `class MulticlassClassificationEvaluator(` * `class MultilabelClassificationEvaluator(` * `class ClusteringEvaluator(` * `class RankingEvaluator(` * `class Binarizer(` * `class BucketedRandomProjectionLSH(` * `class BucketedRandomProjectionLSHModel(` * `class Bucketizer(` * `class ElementwiseProduct(` * `class FeatureHasher(` * `class HashingTF(` * `class _OneHotEncoderParams(` * `class PolynomialExpansion(` * `class QuantileDiscretizer(` * `class _StringIndexerParams(` * `class StopWordsRemover(` * `class VectorAssembler(` * `class VectorSizeHint(` * `class VarianceThresholdSelector(` * `class VarianceThresholdSelectorModel(` * `class UnivariateFeatureSelector(` * `class UnivariateFeatureSelectorModel(` * `class _LinearRegressionParams(` * `class LinearRegressionModel(` * `class IsotonicRegression(` * `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):` * `class DecisionTreeRegressor(` * `class RandomForestRegressor(` * `class _AFTSurvivalRegressionParams(` * `class AFTSurvivalRegression(` * `class AFTSurvivalRegressionModel(` * `class _GeneralizedLinearRegressionParams(` * `class GeneralizedLinearRegression(` * `class GeneralizedLinearRegressionModel(` * `class _FactorizationMachinesParams(` * `class FMRegressionModel(` * `class CrossValidator(` * `class TrainValidationSplit(` * `+ \"class name ` * `class MultivariateGaussian(NamedTuple):` * `class PandasAPIOnSparkAdviceWarning(Warning):` * `class ArrowStreamUDFSerializer(ArrowStreamSerializer):` * `class DayTimeIntervalType(AtomicType):` * `class DayTimeIntervalTypeConverter(object):` * `trait ExpressionBuilder ` * `case class ExpressionStats(expr: Expression)(var useCount: Int)` * `trait PadExpressionBuilderBase extends ExpressionBuilder ` * `case class StringLPad(str: Expression, len: Expression, pad: Expression)` * `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)` * `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)` * `case class PythonMapInArrow(` * `case class DropIndex(` * `public class ColumnIOUtil ` * `case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)` * `case class ParquetColumn(` * `case class DropIndexExec(` * `case class PushedDownOperators(` * `case class TableSampleInfo(` * `trait MapInBatchExec extends UnaryExecNode ` * `case class PythonMapInArrowExec(` * ` // When this is ena
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975146317 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49966/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
LuciferYang commented on a change in pull request #34676: URL: https://github.com/apache/spark/pull/34676#discussion_r753950582 ## File path: pom.xml ## @@ -3580,6 +3580,18 @@ + + mac-on-apple-silicon + + org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest Review comment: Ok ~ So is it necessary for us to remind developers using M1 that they should manually add these tags when testing? I don't think all developers know these at present. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975127306 **[Test build #145495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145495/testReport)** for PR 34596 at commit [`309643b`](https://github.com/apache/spark/commit/309643bea16446d58c9e86e7e0e60988c33934fb). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sadikovi commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
sadikovi commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975124738 @cloud-fan @gengliangwang I updated the code, followed the advice to pass an extra boolean flag `ignoreTimeZone` in the relevant methods. Can you review the changes? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cxzl25 commented on pull request #34493: [SPARK-37217][SQL] The number of dynamic partitions should early check when writing to external tables
cxzl25 commented on pull request #34493: URL: https://github.com/apache/spark/pull/34493#issuecomment-975113863 Can we continue to review this pr? @dongjoon-hyun @sunchao @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
AmplabJenkins removed a comment on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975110769 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49964/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
AmplabJenkins removed a comment on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975110770 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49965/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975110768 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49963/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
AmplabJenkins commented on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975110770 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49965/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
AmplabJenkins commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975110769 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49964/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975110768 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49963/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
SparkQA commented on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975104802 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49965/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
SparkQA commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975102523 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49964/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975102524 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49966/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975083874 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49963/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
HyukjinKwon commented on a change in pull request #34676: URL: https://github.com/apache/spark/pull/34676#discussion_r753929523 ## File path: pom.xml ## @@ -3580,6 +3580,18 @@ + + mac-on-apple-silicon + + org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest Review comment: Hm, I think we don't need to bother with this for now. Testing tags are enough. I would prefer to avoid having profiles here for other combinations of tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33628: [SPARK-36406][CORE] Avoid unnecessary file operations before delete a write failed file held by DiskBlockObjectWriter
SparkQA commented on pull request #33628: URL: https://github.com/apache/spark/pull/33628#issuecomment-975054194 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49965/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
SparkQA commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-975051861 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49964/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34676: [WIP][SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
SparkQA commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-975049090 **[Test build #145494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145494/testReport)** for PR 34676 at commit [`b8e917f`](https://github.com/apache/spark/commit/b8e917fe4d73cf6c8a78fb90764e2a1ebdb250a3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975048158 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975048157 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49962/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments
AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975048157 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49962/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975048158 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuechendi opened a new pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format
xuechendi opened a new pull request #34396: URL: https://github.com/apache/spark/pull/34396 ### What changes were proposed in this pull request? This Jira is aim to support Arrow format in RowToColumnarExec. ### Why are the changes needed? Current ArrowColumnVector is not fully equivalent to OnHeap/OffHeapColumnVector in spark, so RowToColumnarExec doesn't support write to Arrow format so far. since Arrow API is now being more stable, and using pandas udf will perform much better than python udf. ### What has been done in this pull request? I am proposing to support RowToColumnarExec with Arrow. What I did in this PR is to add a load api in ArrowColumnVector to load arrowRecordBatch to ArrowColumnVector, then called inside RowToColumnarExec doExecute. ### How was this patch tested? UTs are also added to test this new API and RowToColumnarExec with ArrowFormat. ### Does this PR introduce _any_ user-facing change? NO Signed-off-by: Chendi Xue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
