[GitHub] [spark] SparkQA removed a comment on pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md
SparkQA removed a comment on pull request #34734: URL: https://github.com/apache/spark/pull/34734#issuecomment-981357525 **[Test build #145700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145700/testReport)** for PR 34734 at commit [`1f24055`](https://github.com/apache/spark/commit/1f24055e2043c521f0ab9a0ae54f039c7c9b26bf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn opened a new pull request #34735: [SPARK-37481][Core] Fix disappearance of skipped stages after they retry
yaooqinn opened a new pull request #34735: URL: https://github.com/apache/spark/pull/34735 ### What changes were proposed in this pull request? When skipped stages retry, their skipped info will be lost on the UI, and then we may see a stage with 200 tasks indeed, shows that it only has 3 tasks but its `retry 1` has 15 tasks and completely different inputs/outputs. Detailed in the Gist here - https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c In this PR, we mark these skipped stages completed and increase the attempId once they get visited. ### Why are the changes needed? fix problems when we have skipped stage retries. ### Does this PR introduce _any_ user-facing change? Yes, the UI will keep the skipped stages info ### How was this patch tested? manually as recorded in https://gist.github.com/yaooqinn/6acb7b74b343a6a6dffe8401f6b7b45c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #34646: [SPARK-37372][K8S] Removing redundant label addition and refactoring related test case
Yikun commented on pull request #34646: URL: https://github.com/apache/spark/pull/34646#issuecomment-981371597 @dongjoon-hyun Would you mind taking a look again? Or I misundertanded your suggestion, it's not enough to update the PR message, I should split this PR to 2 PRs: 1. Remove the redundant edition, then passed new added the ut "assert(executor.pod.getMetadata.getLabels === conf.labels.asJava)", and [existing conf.label ut](https://github.com/apache/spark/blob/a3886ba976469bef0dfafc3da8686a53c5a59d95/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesConfSuite.scala#L157). 2. then refactor CUSTOM_DRIVER_LABELS logic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981369425 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50165/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md
SparkQA commented on pull request #34734: URL: https://github.com/apache/spark/pull/34734#issuecomment-981367713 **[Test build #145700 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145700/testReport)** for PR 34734 at commit [`1f24055`](https://github.com/apache/spark/commit/1f24055e2043c521f0ab9a0ae54f039c7c9b26bf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #34712: [SPARK-37463][SQL] Read/Write Timestamp ntz or ltz to Orc uses UTC timestamp
gengliangwang commented on a change in pull request #34712: URL: https://github.com/apache/spark/pull/34712#discussion_r758099882 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java ## @@ -48,6 +48,9 @@ // The capacity of vectorized batch. private int capacity; + // If the orc file will be read is not write by Spark 3.1 or prior, use UTC timestamp. Review comment: If the Orc file to be read is written by Spark 3.3 or after, use UTC timestamp. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configurations to latest in doc
SparkQA commented on pull request #34734: URL: https://github.com/apache/spark/pull/34734#issuecomment-981357525 **[Test build #145700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145700/testReport)** for PR 34734 at commit [`1f24055`](https://github.com/apache/spark/commit/1f24055e2043c521f0ab9a0ae54f039c7c9b26bf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981357549 **[Test build #145701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145701/testReport)** for PR 34732 at commit [`4598e8b`](https://github.com/apache/spark/commit/4598e8b52e361ef10e0ec3057041b10feb01f8ce). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981357027 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145692/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981357027 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145692/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun opened a new pull request #34734: [SPARK-37480][K8S][DOC] Sync Kubernetes configurations to latest in doc
Yikun opened a new pull request #34734: URL: https://github.com/apache/spark/pull/34734 ### What changes were proposed in this pull request? Sync Kubernetes configurations to latest in doc ### Why are the changes needed? Configurations in docs/running-on-kubernetes.md are not uptodate ### Does this PR introduce _any_ user-facing change? No, docs only ### How was this patch tested? CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981356532 **[Test build #145692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145692/testReport)** for PR 34367 at commit [`354b445`](https://github.com/apache/spark/commit/354b445a7fe645c95bddca0030ad3b56135a0106). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
SparkQA removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981263291 **[Test build #145692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145692/testReport)** for PR 34367 at commit [`354b445`](https://github.com/apache/spark/commit/354b445a7fe645c95bddca0030ad3b56135a0106). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34733: [SPARK-36346][SQL][FOLLOWUP] Rename `withAllOrcReaders` to `withAllNativeOrcReaders`
SparkQA commented on pull request #34733: URL: https://github.com/apache/spark/pull/34733#issuecomment-981351668 **[Test build #145698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145698/testReport)** for PR 34733 at commit [`fc448fc`](https://github.com/apache/spark/commit/fc448fca339ca95a3b3c7e6cd781adf7a119b9ac). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981351647 **[Test build #145699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145699/testReport)** for PR 34732 at commit [`b33d254`](https://github.com/apache/spark/commit/b33d254c8b7cd892e350058645fb8a4d5365cc23). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34726: [SPARK-33875][SQL][FOLLOWUP] Handle the char/varchar column for `Describe column` command
AmplabJenkins removed a comment on pull request #34726: URL: https://github.com/apache/spark/pull/34726#issuecomment-981350686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145688/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981331159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34726: [SPARK-33875][SQL][FOLLOWUP] Handle the char/varchar column for `Describe column` command
AmplabJenkins commented on pull request #34726: URL: https://github.com/apache/spark/pull/34726#issuecomment-981350686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145688/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981350687 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50167/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
gengliangwang commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-981350638 @bersprockets good catch, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AngersZh commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758087493 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: > I think you can just do: > if options is not None: > jsparkSession = self._jvm.SparkSession(self._jsc.sc(), options) If so, we need to add a `if else`, why not just one line. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981348630 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50164/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AngersZh commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758065251 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: With `private[sql] def this(sc: SparkContext) = this(sc, Map.empty)` won't need such changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AngersZh commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758086614 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: > Hm, is `self._jvm.PythonUtils.toScalaMap` required? It's required ``` org.apache.spark.sql.SparkSession([class org.apache.spark.SparkContext, class java.util.HashMap]) does not exist ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Yikun edited a comment on pull request #34717: URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341 Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~ : (, Update: pandas only publish ubuntu wheel after v1.2we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86, yes have the 1.0.1 wheel) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it. and looks like there were some testcase are failed: ``` == ERROR: test_astype (pyspark.pandas.tests.data_type_ops.test_categorical_ops.CategoricalOpsTest) -- Traceback (most recent call last): File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/tests/data_type_ops/test_categorical_ops.py", line 204, in test_astype self.assert_eq(pser.astype(int), psser.astype(int)) File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 224, in assert_eq robj = self._to_pandas(right) File "/Users/jiangyikun/spark/spark/python/pyspark/testing/pandasutils.py", line 245, in _to_pandas return obj.to_pandas() File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1588, in to_pandas return self._to_pandas() File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 1594, in _to_pandas return self._to_internal_pandas().copy() File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/series.py", line 6349, in _to_internal_pandas return self._psdf._internal.to_pandas_frame[self.name] File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/utils.py", line 584, in wrapped_lazy_property setattr(self, attr_name, fn(self)) File "/Users/jiangyikun/spark/spark/python/pyspark/pandas/internal.py", line 1049, in to_pandas_frame pdf = sdf.toPandas() File "/Users/jiangyikun/spark/spark/python/pyspark/sql/pandas/conversion.py", line 185, in toPandas pdf = pd.DataFrame(columns=tmp_column_names).astype( File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 435, in __init__ mgr = init_dict(data, index, columns, dtype=dtype) File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 239, in init_dict val = construct_1d_arraylike_from_scalar(np.nan, len(index), nan_dtype) File "/Users/jiangyikun/venv/lib/python3.8/site-packages/pandas/core/dtypes/cast.py", line 1449, in construct_1d_arraylike_from_scalar dtype = dtype.dtype AttributeError: type object 'object' has no attribute 'dtype' -- ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981343687 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50165/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34726: [SPARK-33875][SQL][FOLLOWUP] Handle the char/varchar column for `Describe column` command
SparkQA removed a comment on pull request #34726: URL: https://github.com/apache/spark/pull/34726#issuecomment-981232745 **[Test build #145688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145688/testReport)** for PR 34726 at commit [`ef74a06`](https://github.com/apache/spark/commit/ef74a06ca0ce87332a75472d3f3120f849d87f5f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34726: [SPARK-33875][SQL][FOLLOWUP] Handle the char/varchar column for `Describe column` command
SparkQA commented on pull request #34726: URL: https://github.com/apache/spark/pull/34726#issuecomment-981342304 **[Test build #145688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145688/testReport)** for PR 34726 at commit [`ef74a06`](https://github.com/apache/spark/commit/ef74a06ca0ce87332a75472d3f3120f849d87f5f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #34717: [SPARK-37465][PYTHON] Bump minimum pandas version to 1.0.0
Yikun edited a comment on pull request #34717: URL: https://github.com/apache/spark/pull/34717#issuecomment-981220341 Sure, thanks for your suggestion, I'd like to update. and I added a simple test to install pandas v1.0.1 ~and run test on https://github.com/apache/spark/pull/34730 , wait for the result.~ : (, Update: pandas only publish ubuntu wheel after v1.2we have to install many deps, otherwise it would be failed when using `pip install pandas==1.0.1`,so I just install in my local env (macos, x86) and running pip install 'pandas==1.0.1' and `python/run-tests --modules=pyspark-pandas,pyspark-pandas-slow --parallelism=2 --python-executable=python3` to test it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
dongjoon-hyun commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-981335424 BTW, thank you, @bersprockets ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981332981 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50167/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34723: [MINOR][SQL] Optimize some Orc test code
dongjoon-hyun commented on pull request #34723: URL: https://github.com/apache/spark/pull/34723#issuecomment-981332407 Let's proceed this after https://github.com/apache/spark/pull/34733 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
dongjoon-hyun commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-981332147 Here is a follow-up PR. - https://github.com/apache/spark/pull/34733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #34733: [SPARK-36346][SQL][FOLLOWUP] Rename withAllOrcReaders to withAllNativeOrcReaders
dongjoon-hyun opened a new pull request #34733: URL: https://github.com/apache/spark/pull/34733 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981325968 **[Test build #145696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981331159 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145696/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981331123 **[Test build #145696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330252 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50166/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330241 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50166/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330252 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50166/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330133 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145697/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981327528 **[Test build #145697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145697/testReport)** for PR 34732 at commit [`f6df6a8`](https://github.com/apache/spark/commit/f6df6a8d4efa1f22dd280396e11b93ddaedd3e68). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330110 **[Test build #145697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145697/testReport)** for PR 34732 at commit [`f6df6a8`](https://github.com/apache/spark/commit/f6df6a8d4efa1f22dd280396e11b93ddaedd3e68). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981330133 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145697/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981329560 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145694/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA removed a comment on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981324141 **[Test build #145694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AmplabJenkins commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981329560 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145694/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981329520 **[Test build #145694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981325968 **[Test build #145696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
HyukjinKwon commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981325158 Thanks for the followup! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
AngersZh commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758065251 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: With `private[sql] def this(sc: SparkContext) = this(sc, Map.empty)` won't need such changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
HyukjinKwon commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758065143 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: I think you can just do: ```python if options is not None: jsparkSession = self._jvm.SparkSession(self._jsc.sc(), options) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
HyukjinKwon commented on a change in pull request #34732: URL: https://github.com/apache/spark/pull/34732#discussion_r758064643 ## File path: python/pyspark/sql/session.py ## @@ -305,10 +305,7 @@ def __init__( ): jsparkSession = self._jvm.SparkSession.getDefaultSession().get() else: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) -if options is not None: -for key, value in options.items(): -jsparkSession.sharedState().conf().set(key, value) +jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)) Review comment: Hm, is `self._jvm.PythonUtils.toScalaMap` required? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
SparkQA commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981324141 **[Test build #145694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981324203 **[Test build #145695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145695/testReport)** for PR 34715 at commit [`758b267`](https://github.com/apache/spark/commit/758b2678030b6939b1a3e26642b09930337f61b5). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
AmplabJenkins removed a comment on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981323452 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50163/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
AmplabJenkins removed a comment on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981323453 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145690/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
HyukjinKwon commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758064015
##
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##
@@ -97,11 +97,11 @@ class SparkSession private(
* since that would cause every new session to reinvoke Spark Session
Extensions on the currently
* running extensions.
*/
- private[sql] def this(sc: SparkContext) = {
+ private[sql] def this(sc: SparkContext, initialSessionOptions: Map[String,
String]) = {
Review comment:
Can we have another constructor: `private[sql] def this(sc:
SparkContext) = this(sc, Map.empty)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
AmplabJenkins commented on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981323453 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145690/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
AmplabJenkins commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981323452 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50163/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
dongjoon-hyun commented on a change in pull request #33588:
URL: https://github.com/apache/spark/pull/33588#discussion_r758063667
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
##
@@ -143,6 +143,13 @@ abstract class OrcTest extends QueryTest with
FileBasedDataSourceTest with Befor
spark.read.orc(file.getAbsolutePath)
}
+ def withAllOrcReaders(code: => Unit): Unit = {
Review comment:
Hi, All.
This naming is misleading because this only tests `native` ORC reader.
Apache Spark provides `hive` ORC reader, too.
Instead of `withAllOrcReaders`, let's use `withAllNativeOrcReaders`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34732: [SPARK-37291][PYSPARK] PySpark create SparkSession should pass initialSessionOptions
AngersZh commented on pull request #34732: URL: https://github.com/apache/spark/pull/34732#issuecomment-981322758 ping @HyukjinKwon @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #34732: [SPARK-37291][PYSPARK] PySpark create SparkSession should pass initialSessionOptions
AngersZh opened a new pull request #34732: URL: https://github.com/apache/spark/pull/34732 ### What changes were proposed in this pull request? In this pr, when create SparkSession, we pass initialSessionOptions to SparkSession, to keep same code path with scala code. ### Why are the changes needed? Keep same code path with scala code. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existed UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #34697: [SPARK-37452][SQL] Char and Varchar break backward compatibility between v3.1 and v2
yaooqinn commented on pull request #34697: URL: https://github.com/apache/spark/pull/34697#issuecomment-981322426 any more concerns from the CCers？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
AngersZh commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981320628 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34679: [SPARK-37437][BUILD] Remove unused hive profile and related CI test
dongjoon-hyun commented on pull request #34679: URL: https://github.com/apache/spark/pull/34679#issuecomment-981318255 +1, late LGTM. Thank you all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34676: [SPARK-37434][BUILD] Add a new profile to auto disable unsupported UTs on MacOs using Apple Silicon
dongjoon-hyun commented on pull request #34676: URL: https://github.com/apache/spark/pull/34676#issuecomment-981316245 Thank you for closing this PR, @LuciferYang . Ya, `leveldb` JNI library is severely outdated while `RocksDB` shows its progress, https://github.com/facebook/rocksdb/issues/7720. Given that, I'm thinking about that we can make Spark `leveldb`-free. Specifically, we may replace `levelDB` part with `RocksDB` part because Spark SQL module already uses `RocksDB` too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
SparkQA commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981313691 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50163/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
SparkQA removed a comment on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981255405 **[Test build #145690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145690/testReport)** for PR 34720 at commit [`2f25efc`](https://github.com/apache/spark/commit/2f25efc47632065bfcdf1baefe1e23d669a3a225). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
SparkQA commented on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981313387 **[Test build #145690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145690/testReport)** for PR 34720 at commit [`2f25efc`](https://github.com/apache/spark/commit/2f25efc47632065bfcdf1baefe1e23d669a3a225). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34620: [SPARK-37209][YARN][TESTS] Fix `YarnShuffleIntegrationSuite` releated UTs when using `hadoop-3.2` profile without `assembly/target/scal
dongjoon-hyun commented on pull request #34620: URL: https://github.com/apache/spark/pull/34620#issuecomment-981311068 Thank you all. +1, late LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34722: [SPARK-37319][K8S][FOLLOWUP] Set JAVA_HOME for Java 17 installed by apt-get
dongjoon-hyun closed pull request #34722: URL: https://github.com/apache/spark/pull/34722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs
HyukjinKwon closed pull request #34685: URL: https://github.com/apache/spark/pull/34685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs
HyukjinKwon commented on pull request #34685: URL: https://github.com/apache/spark/pull/34685#issuecomment-981307601 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
AmplabJenkins removed a comment on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981306552 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145691/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
AmplabJenkins commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981306552 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145691/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA removed a comment on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981255501 **[Test build #145691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145691/testReport)** for PR 34715 at commit [`758b267`](https://github.com/apache/spark/commit/758b2678030b6939b1a3e26642b09930337f61b5). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981306063 **[Test build #145691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145691/testReport)** for PR 34715 at commit [`758b267`](https://github.com/apache/spark/commit/758b2678030b6939b1a3e26642b09930337f61b5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #34691: [SPARK-37447][SQL] Cache LogicalPlan.isStreaming() result in a lazy val
HeartSaVioR commented on pull request #34691: URL: https://github.com/apache/spark/pull/34691#issuecomment-981305889 I can't imagine the case the logical plan somehow replaces the leaf nodes (sources) after other nodes are added on top of leaf nodes. If that is true, I guess this simply works, as the notion of "streaming" is only defined in the leaf nodes. Probably need to double-confirm with experts in SQL area. cc. @cloud-fan @viirya @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34730: [DNM] Trigger test for 'pandas==1.0.1'
AmplabJenkins removed a comment on pull request #34730: URL: https://github.com/apache/spark/pull/34730#issuecomment-981299849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981299850 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50162/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
AmplabJenkins removed a comment on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981299852 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145693/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
AmplabJenkins removed a comment on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981299851 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50161/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
AmplabJenkins removed a comment on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981299853 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50160/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981299850 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50162/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34730: [DNM] Trigger test for 'pandas==1.0.1'
AmplabJenkins commented on pull request #34730: URL: https://github.com/apache/spark/pull/34730#issuecomment-981299849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
AmplabJenkins commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981299851 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50161/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
AmplabJenkins commented on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981299853 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50160/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
AmplabJenkins commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981299852 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145693/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981299164 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50162/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34715: [SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2
SparkQA commented on pull request #34715: URL: https://github.com/apache/spark/pull/34715#issuecomment-981294029 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50161/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34720: [SPARK-37469][WebUI] unified shuffle read block time to shuffle read fetch wait time in StagePage
SparkQA commented on pull request #34720: URL: https://github.com/apache/spark/pull/34720#issuecomment-981293088 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50160/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
SparkQA commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981292258 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50163/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34730: [DNM] Trigger test for 'pandas==1.0.1'
SparkQA commented on pull request #34730: URL: https://github.com/apache/spark/pull/34730#issuecomment-981291980 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50159/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34730: [DNM] Trigger test for 'pandas==1.0.1'
SparkQA removed a comment on pull request #34730: URL: https://github.com/apache/spark/pull/34730#issuecomment-981232709 **[Test build #145687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145687/testReport)** for PR 34730 at commit [`ca77e73`](https://github.com/apache/spark/commit/ca77e7363d1c3542612692778e0248d480074ea6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
HyukjinKwon commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981287828 Let's get https://github.com/apache/spark/pull/34685 done first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34730: [DNM] Trigger test for 'pandas==1.0.1'
SparkQA commented on pull request #34730: URL: https://github.com/apache/spark/pull/34730#issuecomment-981287686 **[Test build #145687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145687/testReport)** for PR 34730 at commit [`ca77e73`](https://github.com/apache/spark/commit/ca77e7363d1c3542612692778e0248d480074ea6). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
SparkQA removed a comment on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981279640 **[Test build #145693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145693/testReport)** for PR 34731 at commit [`b947cc4`](https://github.com/apache/spark/commit/b947cc4ef9f3ae2b15a89cc6090b7408cf6f3392). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34731: [SPARK-37153][PYTHON] Inline type hints for python/pyspark/profiler.py
SparkQA commented on pull request #34731: URL: https://github.com/apache/spark/pull/34731#issuecomment-981286377 **[Test build #145693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145693/testReport)** for PR 34731 at commit [`b947cc4`](https://github.com/apache/spark/commit/b947cc4ef9f3ae2b15a89cc6090b7408cf6f3392). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-981282705 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50162/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
