Xuebin Su has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24674
Change subject: IMPALA-15252: Add Python UDF support ...................................................................... IMPALA-15252: Add Python UDF support This patch adds support for creating and calling Python UDFs. To execute Python code, - the static library `libpython3.*.a` is linked to impalad, and - a Python interpreter is initialized inside the impalad process. The feature is available only if the impalad flag `--python_executable_path` is set, of which the value is required to initialize the Python interpreter. For better performance, Python UDF calls are evaluated on one batch at a time, instead of on one row. Values in one column of one RowBatch is converted into an Apache Arrow array before being passed to the Python function as an argument. It is required to add the path containing the PyArrow library to Python's module search path `sys.path` in the `*.py` file containing the Python function. See `be/src/exprs/python-udf-call.h` for more details about how Python UDFs work. Batched evaluation is controlled by query option `enable_batched_evaluation`. If it is set to false, calling any Python UDF in the query will cause an error. In this patch, batched evaluation support is only added for the following kinds of `ScalarExpr`: - PythonUdfCall, - SlotRef, and - Literal and the following kinds of `ExecNode`: - UnionNode when it has only constant expressions, - BufferedPlanRootSink, - NonGroupingAggregator, and - in FeSupport_NativeEvalExprsWithoutRow() See `be/src/exprs/scalar-expr.h` for more details about how batched evaluation works. Testing: - Added E2E tests. Change-Id: I04207ac53a53381b0dbdb9a7768665fb95aad519 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/codegen/gen_ir_descriptions.py A be/src/common/status-or.h M be/src/exec/aggregation-node-base.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/union-node.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/agg-fn-evaluator.cc M be/src/exprs/agg-fn-evaluator.h M be/src/exprs/agg-fn.cc M be/src/exprs/expr-value.h M be/src/exprs/expr.h M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/exprs/literal.cc M be/src/exprs/literal.h A be/src/exprs/python-udf-call.cc A be/src/exprs/python-udf-call.h M be/src/exprs/scalar-expr-evaluator-ir.cc M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h A be/src/exprs/udf-call.h M be/src/runtime/descriptors.cc M be/src/runtime/fragment-state.h M be/src/runtime/lib-cache.cc M be/src/runtime/lib-cache.h M be/src/runtime/row-batch.h M be/src/runtime/tuple.h M be/src/service/fe-support.cc M be/src/service/impalad-main.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare-test.cc A cmake_modules/FindPython.cmake M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/catalog/Db.java A testdata/udfs/impala_python_udfs.py A testdata/udfs/impala_python_udfs_no_pyarrow.py A testdata/workloads/functional-query/queries/QueryTest/python-udfs-no-pyarrow.test A testdata/workloads/functional-query/queries/QueryTest/python-udfs.test M tests/custom_cluster/test_permanent_udfs.py 52 files changed, 1,573 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/24674/2 -- To view, visit http://gerrit.cloudera.org:8080/24674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I04207ac53a53381b0dbdb9a7768665fb95aad519 Gerrit-Change-Number: 24674 Gerrit-PatchSet: 2 Gerrit-Owner: Xuebin Su <[email protected]>
