ueshin commented on a change in pull request #29686:
URL: https://github.com/apache/spark/pull/29686#discussion_r485174089
##########
File path: pom.xml
##########
@@ -2329,19 +2329,26 @@
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
+ <artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
- <artifactId>netty-buffer</artifactId>
+ <artifactId>netty-common</artifactId>
</exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-netty</artifactId>
Review comment:
Don't we need any configs except for adding the dependency to indicate
which allocator should be used?
##########
File path: python/pyspark/sql/pandas/utils.py
##########
@@ -40,7 +40,7 @@ def require_minimum_pyarrow_version():
""" Raise ImportError if minimum version of pyarrow is not installed
"""
# TODO(HyukjinKwon): Relocate and deduplicate the version specification.
- minimum_pyarrow_version = "0.15.1"
+ minimum_pyarrow_version = "1.0.0"
Review comment:
+1
##########
File path: python/setup.py
##########
@@ -96,11 +96,11 @@ def _supports_symlinks():
file=sys.stderr)
sys.exit(-1)
-# If you are changing the versions here, please also change
./python/pyspark/sql/utils.py
+# If you are changing the versions here, please also change
./python/pyspark/sql/pandas/utils.py
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
