LuciferYang commented on code in PR #37843:
URL: https://github.com/apache/spark/pull/37843#discussion_r972590525
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Expression.java:
##########
@@ -44,7 +46,12 @@ public interface Expression {
* List of fields or columns that are referenced by this expression.
*/
default NamedReference[] references() {
- return Arrays.stream(children()).map(e -> e.references())
- .flatMap(Arrays::stream).distinct().toArray(NamedReference[]::new);
+ // SPARK-40398: Replace `Arrays.stream()...distinct()`
+ // to this for perf gain, the result order is not important.
+ Set<NamedReference> set = new HashSet<>();
+ for (Expression e : children()) {
+ Collections.addAll(set, e.references());
+ }
+ return set.toArray(new NamedReference[0]);
Review Comment:
`nstall dependencies for documentation generation` still failed
https://github.com/LuciferYang/spark/actions/runs/3064658259/jobs/4947983207
```
× Building wheel for pyzmq (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [186 lines of output]
/tmp/pip-build-env-812k46kb/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:262:
UserWarning: Unknown distribution option: 'cffi_modules'
warnings.warn(msg)
running bdist_wheel
running build
running build_py
copying zmq/error.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/asyncio.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/_future.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/_typing.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/decorators.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/constants.py -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/__init__.py -> build/lib.linux-x86_64-cpython-39/zmq
creating build/lib.linux-x86_64-cpython-39/zmq/log
copying zmq/log/__main__.py ->
build/lib.linux-x86_64-cpython-39/zmq/log
copying zmq/log/handlers.py ->
build/lib.linux-x86_64-cpython-39/zmq/log
copying zmq/log/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/log
creating build/lib.linux-x86_64-cpython-39/zmq/green
copying zmq/green/device.py ->
build/lib.linux-x86_64-cpython-39/zmq/green
copying zmq/green/poll.py ->
build/lib.linux-x86_64-cpython-39/zmq/green
copying zmq/green/core.py ->
build/lib.linux-x86_64-cpython-39/zmq/green
copying zmq/green/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/green
creating build/lib.linux-x86_64-cpython-39/zmq/green/eventloop
copying zmq/green/eventloop/ioloop.py ->
build/lib.linux-x86_64-cpython-39/zmq/green/eventloop
copying zmq/green/eventloop/zmqstream.py ->
build/lib.linux-x86_64-cpython-39/zmq/green/eventloop
copying zmq/green/eventloop/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/green/eventloop
creating build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_imports.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_future.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_asyncio.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_win32_shim.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_etc.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_draft.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_message.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_monqueue.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_multipart.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_error.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_constants.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_poll.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/conftest.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_monitor.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_security.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_context.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_mypy.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_ssh.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_z85.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_cffi_backend.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_ioloop.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_auth.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_pair.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_device.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_includes.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_socket.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_cython.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_log.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_pubsub.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_ext.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_zmqstream.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_reqrep.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_proxy_steerable.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_version.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_retry_eintr.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
copying zmq/tests/test_decorators.py ->
build/lib.linux-x86_64-cpython-39/zmq/tests
creating build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/proxydevice.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/monitoredqueuedevice.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/proxysteerabledevice.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/monitoredqueue.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/basedevice.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/devices/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/devices
creating build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/asyncio.py ->
build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/certs.py -> build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/ioloop.py ->
build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/base.py -> build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/auth
copying zmq/auth/thread.py ->
build/lib.linux-x86_64-cpython-39/zmq/auth
creating build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/interop.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/garbage.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/win32.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/jsonapi.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/monitor.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/z85.py -> build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/strtypes.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/utils
creating build/lib.linux-x86_64-cpython-39/zmq/backend
copying zmq/backend/select.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend
copying zmq/backend/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend
creating build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
creating build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/error.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/context.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/message.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/_poll.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/devices.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/socket.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/utils.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/backend/cffi/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
creating build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/version.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/tracker.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/context.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/attrsettr.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/poll.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/frame.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/stopwatch.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/socket.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
copying zmq/sugar/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
creating build/lib.linux-x86_64-cpython-39/zmq/ssh
copying zmq/ssh/tunnel.py -> build/lib.linux-x86_64-cpython-39/zmq/ssh
copying zmq/ssh/forward.py -> build/lib.linux-x86_64-cpython-39/zmq/ssh
copying zmq/ssh/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/ssh
creating build/lib.linux-x86_64-cpython-39/zmq/eventloop
copying zmq/eventloop/ioloop.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop
copying zmq/eventloop/zmqstream.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop
copying zmq/eventloop/future.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop
copying zmq/eventloop/_deprecated.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop
copying zmq/eventloop/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop
creating build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/log.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/stack_context.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/concurrent.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/ioloop.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/util.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
copying zmq/eventloop/minitornado/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado
creating
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/interface.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/auto.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/common.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/windows.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/posix.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/eventloop/minitornado/platform/__init__.py ->
build/lib.linux-x86_64-cpython-39/zmq/eventloop/minitornado/platform
copying zmq/__init__.pxd -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/__init__.pyi -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/py.typed -> build/lib.linux-x86_64-cpython-39/zmq
copying zmq/devices/monitoredqueue.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/devices
copying zmq/utils/buffers.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/zmq_compat.h ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/pyversion_compat.h ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/mutex.h ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/ipcmaxlen.h ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/getpid_compat.h ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/compiler.json ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/utils/config.json ->
build/lib.linux-x86_64-cpython-39/zmq/utils
copying zmq/backend/__init__.pyi ->
build/lib.linux-x86_64-cpython-39/zmq/backend
copying zmq/backend/cython/context.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/socket.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/__init__.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/message.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/checkrc.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/libzmq.pxd ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cython/constant_enums.pxi ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cython
copying zmq/backend/cffi/_cdefs.h ->
build/lib.linux-x86_64-cpython-39/zmq/backend/cffi
copying zmq/sugar/__init__.pyi ->
build/lib.linux-x86_64-cpython-39/zmq/sugar
running build_ext
running configure
Using bundled libzmq
already have bundled/zeromq
already have platform.hpp
checking for timer_create
************************************************
************************************************
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare
-DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat
-Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.9 -c
/tmp/timer_createwdqm_dcl.c -o tmp/timer_createwdqm_dcl.o
/tmp/timer_createwdqm_dcl.c: In function ‘main’:
/tmp/timer_createwdqm_dcl.c:2:5: warning: implicit declaration of
function ‘timer_create’ [-Wimplicit-function-declaration]
2 | timer_create();
| ^~~~~~~~~~~~
x86_64-linux-gnu-gcc -pthread tmp/timer_createwdqm_dcl.o
-L/usr/lib/x86_64-linux-gnu -o a.out
/usr/bin/ld: tmp/timer_createwdqm_dcl.o: in function `main':
/tmp/timer_createwdqm_dcl.c:2: undefined reference to `timer_create'
collect2: error: ld returned 1 exit status
no timer_create, linking librt
************************************************
building 'zmq.libzmq' extension
creating build/temp.linux-x86_64-cpython-39/buildutils
creating build/temp.linux-x86_64-cpython-39/bundled
creating build/temp.linux-x86_64-cpython-39/bundled/zeromq
creating build/temp.linux-x86_64-cpython-39/bundled/zeromq/src
x86_64-linux-gnu-g++ -pthread -std=c++11 -pthread -Wno-unused-result
-Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong
-Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DZMQ_HAVE_CURVE=1
-DZMQ_USE_TWEETNACL=1 -DZMQ_USE_EPOLL=1 -DZMQ_IOTHREADS_USE_EPOLL=1
-DZMQ_POLL_BASED_ON_POLL=1 -Ibundled/zeromq/include -Ibundled
-I/usr/include/python3.9 -c buildutils/initlibzmq.cpp -o
build/temp.linux-x86_64-cpython-39/buildutils/initlibzmq.o
buildutils/initlibzmq.cpp:10:10: fatal error: Python.h: No such file
or directory
10 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem
with pip.
ERROR: Failed building wheel for pyzmq
ERROR: Could not build wheels for pyzmq, which is required to install
pyproject.toml-based projects
Failed to build pyzmq
Error: Process completed with exit code 1.
```
friendly ping @HyukjinKwon @Yikun to help to check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
