Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
dongjoon-hyun commented on PR #44077: URL: https://github.com/apache/spark/pull/44077#issuecomment-1832478448 Thank you for your decision, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
LuciferYang closed pull request #44077: [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 URL: https://github.com/apache/spark/pull/44077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
LuciferYang commented on PR #44077: URL: https://github.com/apache/spark/pull/44077#issuecomment-1832467193 After carefully reading https://lists.apache.org/thread/mrx0y078cf3ozs7czykvv864y6dr55xq, I have decided to abandon the deletion of HiveContext. As @gatorsmile said, its maintenance cost is not high, but it will increase the cost of users migrating to Spark4.0, so I also believe it is not worth deleting it in this context. I will close this pr. Thanks @dongjoon-hyun @gatorsmile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
LuciferYang commented on code in PR #44077: URL: https://github.com/apache/spark/pull/44077#discussion_r1409591501 ## python/pyspark/sql/context.py: ## @@ -311,6 +312,24 @@ def registerJavaFunction( ) return self.sparkSession.udf.registerJavaFunction(name, javaClassName, returnType) +# TODO(andrew): delete this once we refactor things to take in SparkSession +def _inferSchema(self, rdd: RDD, samplingRatio: Optional[float] = None) -> StructType: Review Comment: Thanks @dongjoon-hyun, the code in this file should be okay now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
dongjoon-hyun commented on PR #44077: URL: https://github.com/apache/spark/pull/44077#issuecomment-1832316270 I agree with you, @LuciferYang . Thank you for the heads-up email. Since we need an official vote for this specific API, I replied your email thread. Let's see the community atmosphere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
LuciferYang commented on code in PR #44077: URL: https://github.com/apache/spark/pull/44077#discussion_r1409563409 ## python/pyspark/sql/context.py: ## @@ -311,6 +312,24 @@ def registerJavaFunction( ) return self.sparkSession.udf.registerJavaFunction(name, javaClassName, returnType) +# TODO(andrew): delete this once we refactor things to take in SparkSession +def _inferSchema(self, rdd: RDD, samplingRatio: Optional[float] = None) -> StructType: Review Comment: Sorry, I think I made a mistake during the rebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
dongjoon-hyun commented on code in PR #44077: URL: https://github.com/apache/spark/pull/44077#discussion_r1409558060 ## python/pyspark/sql/context.py: ## @@ -311,6 +312,24 @@ def registerJavaFunction( ) return self.sparkSession.udf.registerJavaFunction(name, javaClassName, returnType) +# TODO(andrew): delete this once we refactor things to take in SparkSession +def _inferSchema(self, rdd: RDD, samplingRatio: Optional[float] = None) -> StructType: Review Comment: What is this addition? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-46171][SQL][PYTHON][R][DOCS] Remove `HiveContext` from Apache Spark 4.0 [spark]
LuciferYang commented on PR #44077: URL: https://github.com/apache/spark/pull/44077#issuecomment-1831679067 Already send a email to dev mail list for discussion https://lists.apache.org/thread/qfznmh1dvjf9r3qn2qc8zkryk3x1t05w -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
