[jira] [Commented] (FLINK-32604) PyFlink end-to-end fails with kafka-server-stop.sh: No such file or directory
[
https://issues.apache.org/jira/browse/FLINK-32604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747430#comment-17747430
]
Huang Xingbo commented on FLINK-32604:
--
It is a downloading error. We can keep observing the frequency of this failed
case.
> PyFlink end-to-end fails with kafka-server-stop.sh: No such file or
> directory
> ---
>
> Key: FLINK-32604
> URL: https://issues.apache.org/jira/browse/FLINK-32604
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51253=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=7883
> fails as
> {noformat}
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/kafka-common.sh: line
> 117:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-27379214502/kafka_2.12-3.2.3/bin/kafka-server-stop.sh:
> No such file or directory
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/kafka-common.sh: line
> 121:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-27379214502/kafka_2.12-3.2.3/bin/zookeeper-server-stop.sh:
> No such file or directory
> Jul 13 19:43:07 [FAIL] Test script contains errors.
> Jul 13 19:43:07 Checking of logs skipped.
> Jul 13 19:43:07
> Jul 13 19:43:07 [FAIL] 'PyFlink end-to-end test' failed after 0 minutes and
> 40 seconds! Test exited with exit code 1
> Jul 13 19:43:07
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-31099) Chained WindowOperator throws NPE in PyFlink ThreadMode
[
https://issues.apache.org/jira/browse/FLINK-31099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo resolved FLINK-31099.
--
Resolution: Fixed
Merged into master via ca770b3d905936d8a93071210bd6542b6733221d
Merged into release-1.17 via c7c035a2413c04cd75948d8364e0770b97499901
Merged into release-1.16 via e3c0060e7fca53e0e01cb91e00607c8146b85604
> Chained WindowOperator throws NPE in PyFlink ThreadMode
> ---
>
> Key: FLINK-31099
> URL: https://issues.apache.org/jira/browse/FLINK-31099
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.2
>
>
> Test case
> {code:python}
> config = Configuration()
> config.set_string("python.execution-mode", "process")
> env = StreamExecutionEnvironment.get_execution_environment(config)
> class MyTimestampAssigner(TimestampAssigner, ABC):
> def extract_timestamp(self, value: tuple, record_timestamp: int) -> int:
> return value[0]
> ds = env.from_collection(
> [(167646168, "a1", "b1", 1), (167646168, "a1", "b1", 1),
> (167646168, "a2", "b2", 1), (167646168, "a1", "b2", 1),
> (167646174, "a1", "b1", 1), (167646174, "a2", "b2", 1)]
> ).assign_timestamps_and_watermarks(
>
> WatermarkStrategy.for_monotonous_timestamps().with_timestamp_assigner(MyTimestampAssigner())
> )
> ds.key_by(
> lambda x: (x[0], x[1], x[2])
> ).window(
> TumblingEventTimeWindows.of(Time.minutes(1))
> ).reduce(
> lambda x, y: (x[0], x[1], x[2], x[3] + y[3]),
> output_type=Types.TUPLE([Types.LONG(), Types.STRING(), Types.STRING(),
> Types.INT()])
> # ).filter(
> # lambda x: x[1] == "a1"
> ).map(
> lambda x: (x[0], x[1], x[3]),
> output_type=Types.TUPLE([Types.LONG(), Types.STRING(), Types.INT()])
> ).print()
> env.execute()
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31099) Chained WindowOperator throws NPE in PyFlink ThreadMode
Huang Xingbo created FLINK-31099:
Summary: Chained WindowOperator throws NPE in PyFlink ThreadMode
Key: FLINK-31099
URL: https://issues.apache.org/jira/browse/FLINK-31099
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.16.1, 1.17.0
Reporter: Huang Xingbo
Assignee: Huang Xingbo
Fix For: 1.17.0, 1.16.2
Test case
{code:python}
config = Configuration()
config.set_string("python.execution-mode", "process")
env = StreamExecutionEnvironment.get_execution_environment(config)
class MyTimestampAssigner(TimestampAssigner, ABC):
def extract_timestamp(self, value: tuple, record_timestamp: int) -> int:
return value[0]
ds = env.from_collection(
[(167646168, "a1", "b1", 1), (167646168, "a1", "b1", 1),
(167646168, "a2", "b2", 1), (167646168, "a1", "b2", 1),
(167646174, "a1", "b1", 1), (167646174, "a2", "b2", 1)]
).assign_timestamps_and_watermarks(
WatermarkStrategy.for_monotonous_timestamps().with_timestamp_assigner(MyTimestampAssigner())
)
ds.key_by(
lambda x: (x[0], x[1], x[2])
).window(
TumblingEventTimeWindows.of(Time.minutes(1))
).reduce(
lambda x, y: (x[0], x[1], x[2], x[3] + y[3]),
output_type=Types.TUPLE([Types.LONG(), Types.STRING(), Types.STRING(),
Types.INT()])
# ).filter(
# lambda x: x[1] == "a1"
).map(
lambda x: (x[0], x[1], x[3]),
output_type=Types.TUPLE([Types.LONG(), Types.STRING(), Types.INT()])
).print()
env.execute()
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo resolved FLINK-28786.
--
Resolution: Fixed
Merged into master via 1e8d528e293da7ab990f1406fcd209ff4fb177b3
Merged into release-1.17 via 556a3faae9f147edf1c263e57a6661fc88b8f7ab
Merged into release-1.16 via 1e1461945d633486ee41f7137743837d0756e9b6
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.2
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-28786:
-
Fix Version/s: 1.16.2
(was: 1.16.1)
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.2
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reopened FLINK-28786:
--
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.1
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685164#comment-17685164
]
Huang Xingbo edited comment on FLINK-28786 at 2/7/23 9:12 AM:
--
If you encounter the following stack trace in m1
{code:java}
:219: RuntimeWarning:
apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary
incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 129, in
word_count(known_args.input, known_args.output)
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 49, in word_count
t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 121, in create
return TableEnvironment(j_tenv)
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 100, in __init__
self._open()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1637, in _open
startup_loopback_server()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1628, in startup_loopback_server
from pyflink.fn_execution.beam.beam_worker_pool_service import \
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
line 44, in
from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
pylint: disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
line 21, in
import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
line 27, in
from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
RemoteOperatorStateBackend
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
line 33, in
from pyflink.fn_execution.beam.beam_coders import FlinkCoder
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
line 27, in
from pyflink.fn_execution.beam import beam_coder_impl_fast as
beam_coder_impl
File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
pyflink.fn_execution.beam.beam_coder_impl_fast
KeyError: '__pyx_vtable__'
{code}
you can execute the following command to solve the problem
{code:java}
pip install cython==0.29.24
brew install gcc
pip uninstall apache-flink
pip uninstall apache-beam
pip install apache-flink
{code}
was (Author: hxbks2ks):
If you encounter the following stack trace in m1
{code:java}
:219: RuntimeWarning:
apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary
incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 129, in
word_count(known_args.input, known_args.output)
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 49, in word_count
t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 121, in create
return TableEnvironment(j_tenv)
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 100, in __init__
self._open()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1637, in _open
startup_loopback_server()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1628, in startup_loopback_server
from pyflink.fn_execution.beam.beam_worker_pool_service import \
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
line 44, in
from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
pylint: disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
line 21, in
import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
line 27, in
from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
RemoteOperatorStateBackend
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
line 33, in
from pyflink.fn_execution.beam.beam_coders import
[jira] [Comment Edited] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685164#comment-17685164
]
Huang Xingbo edited comment on FLINK-28786 at 2/7/23 9:11 AM:
--
If you encounter the following stack trace in m1
{code:java}
:219: RuntimeWarning:
apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary
incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 129, in
word_count(known_args.input, known_args.output)
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 49, in word_count
t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 121, in create
return TableEnvironment(j_tenv)
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 100, in __init__
self._open()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1637, in _open
startup_loopback_server()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1628, in startup_loopback_server
from pyflink.fn_execution.beam.beam_worker_pool_service import \
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
line 44, in
from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
pylint: disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
line 21, in
import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
line 27, in
from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
RemoteOperatorStateBackend
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
line 33, in
from pyflink.fn_execution.beam.beam_coders import FlinkCoder
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
line 27, in
from pyflink.fn_execution.beam import beam_coder_impl_fast as
beam_coder_impl
File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
pyflink.fn_execution.beam.beam_coder_impl_fast
KeyError: '__pyx_vtable__'
{code}
you can execute the following command to solve the problem
{code:java}
pip install cython==0.29.24
brew install gcc
pip uninstall apache-flink
pip install apache-flink
{code}
was (Author: hxbks2ks):
If you encounter the following stack trace in m1
{code:java}
:219: RuntimeWarning:
apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary
incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 129, in
word_count(known_args.input, known_args.output)
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 49, in word_count
t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 121, in create
return TableEnvironment(j_tenv)
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 100, in __init__
self._open()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1637, in _open
startup_loopback_server()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1628, in startup_loopback_server
from pyflink.fn_execution.beam.beam_worker_pool_service import \
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
line 44, in
from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
pylint: disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
line 21, in
import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
line 27, in
from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
RemoteOperatorStateBackend
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
line 33, in
from pyflink.fn_execution.beam.beam_coders import FlinkCoder
File
[jira] [Commented] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685164#comment-17685164
]
Huang Xingbo commented on FLINK-28786:
--
If you encounter the following stack trace in m1
{code:java}
:219: RuntimeWarning:
apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary
incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 129, in
word_count(known_args.input, known_args.output)
File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
line 49, in word_count
t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 121, in create
return TableEnvironment(j_tenv)
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 100, in __init__
self._open()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1637, in _open
startup_loopback_server()
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
line 1628, in startup_loopback_server
from pyflink.fn_execution.beam.beam_worker_pool_service import \
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
line 44, in
from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
pylint: disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
line 21, in
import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
disable=unused-import
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
line 27, in
from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
RemoteOperatorStateBackend
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
line 33, in
from pyflink.fn_execution.beam.beam_coders import FlinkCoder
File
"/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
line 27, in
from pyflink.fn_execution.beam import beam_coder_impl_fast as
beam_coder_impl
File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
pyflink.fn_execution.beam.beam_coder_impl_fast
KeyError: '__pyx_vtable__'
{code}
you can execute the following command to solve the problem
{code:java}
cython==0.29.24
brew install gcc
pip uninstall apache-flink
pip install apache-flink
{code}
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.1
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
>
[jira] [Closed] (FLINK-29421) Support python 3.10
[
https://issues.apache.org/jira/browse/FLINK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-29421.
Resolution: Fixed
Merged into master via 838b79f5b9cc1a4cf253b2c17009f337bf569ecc
> Support python 3.10
> ---
>
> Key: FLINK-29421
> URL: https://issues.apache.org/jira/browse/FLINK-29421
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
>Reporter: Eric Sirianni
>Assignee: Huang Xingbo
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> The {{apache-flink}} package fails to install on Python 3.10 due to inability
> to compile {{numpy}}
> {noformat}
> numpy/core/src/multiarray/scalartypes.c.src:3242:12: error: too
> few arguments to function ‘_Py_HashDouble’
> 3242 | return
> _Py_HashDouble(npy_half_to_double(((PyHalfScalarObject *)obj)->obval));
> |^~
> In file included from
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/Python.h:77,
> from
> numpy/core/src/multiarray/scalartypes.c.src:3:
>
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/pyhash.h:10:23:
> note: declared here
>10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
> {noformat}
> Numpy issue https://github.com/numpy/numpy/issues/19033
> [Mailing list
> thread|https://lists.apache.org/thread/f4r9hjt1l33xf5ngnswszhnls4cxkk52]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30753) Py4J cannot acquire Table.explain() method
[ https://issues.apache.org/jira/browse/FLINK-30753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-30753: - Affects Version/s: 1.17.0 (was: 1.16.0) > Py4J cannot acquire Table.explain() method > -- > > Key: FLINK-30753 > URL: https://issues.apache.org/jira/browse/FLINK-30753 > Project: Flink > Issue Type: Technical Debt > Components: API / Python >Affects Versions: 1.17.0 >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45044=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30753) Py4J cannot acquire Table.explain() method
[ https://issues.apache.org/jira/browse/FLINK-30753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-30753. Fix Version/s: 1.17.0 Assignee: Yunfeng Zhou Resolution: Fixed Merged into master via 9ebb331be45dc2b094aca215e1062dad9800cd76 > Py4J cannot acquire Table.explain() method > -- > > Key: FLINK-30753 > URL: https://issues.apache.org/jira/browse/FLINK-30753 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45044=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30753) Py4J cannot acquire Table.explain() method
[ https://issues.apache.org/jira/browse/FLINK-30753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-30753: - Issue Type: Technical Debt (was: Improvement) > Py4J cannot acquire Table.explain() method > -- > > Key: FLINK-30753 > URL: https://issues.apache.org/jira/browse/FLINK-30753 > Project: Flink > Issue Type: Technical Debt > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45044=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30191) Update py4j from 0.10.9.3 to 0.10.9.7
[ https://issues.apache.org/jira/browse/FLINK-30191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-30191. Fix Version/s: 1.17.0 Resolution: Done Merged into master via 0bbc7b1e9fed89b8c3e8ec67b7b0dad5999c2c01 > Update py4j from 0.10.9.3 to 0.10.9.7 > - > > Key: FLINK-30191 > URL: https://issues.apache.org/jira/browse/FLINK-30191 > Project: Flink > Issue Type: Technical Debt > Components: API / Python >Reporter: Martijn Visser >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29000) Support python UDF in the SQL Gateway
[ https://issues.apache.org/jira/browse/FLINK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29000: - Fix Version/s: 1.17.0 > Support python UDF in the SQL Gateway > - > > Key: FLINK-29000 > URL: https://issues.apache.org/jira/browse/FLINK-29000 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Table SQL / Gateway >Affects Versions: 1.17.0 >Reporter: Shengkai Fang >Assignee: Xingbo Huang >Priority: Major > Fix For: 1.17.0 > > > Currently Flink SQL Client supports python UDF, the Gateway should also > support this feature if the SQL Client is able to submit SQL to the Gateway. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-27716) Add Python API docs in ML
[ https://issues.apache.org/jira/browse/FLINK-27716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo reassigned FLINK-27716: Assignee: Jiang Xin > Add Python API docs in ML > - > > Key: FLINK-27716 > URL: https://issues.apache.org/jira/browse/FLINK-27716 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation, Library / Machine Learning >Reporter: Huang Xingbo >Assignee: Jiang Xin >Priority: Major > Fix For: ml-2.2.0 > > > We can use sphinx same as pyflink or other tools to generate Python API docs > of ML -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30619) AdaptiveSchedulerTest.testStatusMetrics is not stable
[
https://issues.apache.org/jira/browse/FLINK-30619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-30619:
-
Affects Version/s: 1.17.0
> AdaptiveSchedulerTest.testStatusMetrics is not stable
> -
>
> Key: FLINK-30619
> URL: https://issues.apache.org/jira/browse/FLINK-30619
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We experience a test instability in
> {{AdaptiveSchedulerTest.testStatusMetrics}}.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44635=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7c1d86e3-35bd-5fd5-3b7c-30c126a78702=8475
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30619) AdaptiveSchedulerTest.testStatusMetrics is not stable
[
https://issues.apache.org/jira/browse/FLINK-30619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677690#comment-17677690
]
Huang Xingbo commented on FLINK-30619:
--
same case in master:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44951=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
> AdaptiveSchedulerTest.testStatusMetrics is not stable
> -
>
> Key: FLINK-30619
> URL: https://issues.apache.org/jira/browse/FLINK-30619
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We experience a test instability in
> {{AdaptiveSchedulerTest.testStatusMetrics}}.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44635=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7c1d86e3-35bd-5fd5-3b7c-30c126a78702=8475
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-30637) In linux-aarch64 environment, using “is” judgment to match the window type of overwindow have returned incorrect matching results
[
https://issues.apache.org/jira/browse/FLINK-30637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo resolved FLINK-30637.
--
Fix Version/s: 1.16.1
1.15.4
Assignee: Xin Chen
Resolution: Fixed
Merged into master via d053867fb5c0fc647ea9266aab35598d7f3fc5c4
Merged into release-1.16 via eca940c5bf9e17c90dbb6f35e4ba370027137368
Merged into release-1.15 via 4035d61a2756ec16046fb687f533be0501fbbd35
> In linux-aarch64 environment, using “is” judgment to match the window type of
> overwindow have returned incorrect matching results
> -
>
> Key: FLINK-30637
> URL: https://issues.apache.org/jira/browse/FLINK-30637
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.6
> Environment: Linux version 5.10.0-60.18.0.50.oe2203.aarch64
> (abuild@obs-worker-002) (gcc_old (GCC) 10.3.1, GNU ld (GNU Binutils) 2.37) #1
> SMP Wed Mar 30 02:43:08 UTC 2022
>
> pyflink-version：1.13.6
>Reporter: Xin Chen
>Assignee: Xin Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.1, 1.15.4
>
>
> In linux-arch64 environment, “window_type is
> OverWindow.ROW_UNBOUNDED_FOLLOWING” in in the
> PandasBatchOverWindowAggregateFunctionOperation class of the pyflink source
> code has returned the wrong result.
> For example, when window_type is 6, it represents the window type of
> ‘ROW_UNBOUNDED_FOLLOWING’, but “window_type is
> OverWindow.ROW_UNBOUNDED_FOLLOWING” return false because the memory address
> of window_type has changed. It will lead to the wrong type of window, such as
> row sliding window, so, the wrong input data of python udf have been
> assembled and wrong results of that have appeared.
>
> Specifically, the pyflink unit testcase is
> ‘test_over_window_aggregate_function’ in
> ‘pyflink\table\tests\test_pandas_udaf.py’. It performance incorrectly when I
> execute it by pytest on linux-aarch64 system. I cut this unit use case to the
> following code and executed it in the flink standalone mode of aarch64
> system, and got the same error result:
>
> {code:java}
> import unittest
> from pyflink.table import DataTypes, TableEnvironment, EnvironmentSettings
> from pyflink.table.udf import udaf, AggregateFunction
> class MaxAdd(AggregateFunction, unittest.TestCase):
> def open(self, function_context):
> mg = function_context.get_metric_group()
> self.counter = mg.add_group("key", "value").counter("my_counter")
> self.counter_sum = 0
> def get_value(self, accumulator):
> # counter
> self.counter.inc(10)
> self.counter_sum += 10
> return accumulator[0]
> def create_accumulator(self):
> return []
> def accumulate(self, accumulator, *args):
> result = 0
> for arg in args:
> result += arg.max()
> accumulator.append(result)
> @udaf(result_type=DataTypes.FLOAT(), func_type="pandas")
> def mean_udaf(v):
> import logging
> logging.error("debug")
> logging.error(v)
> return v.mean()
> t_env = TableEnvironment.create(
>
> EnvironmentSettings.new_instance().in_batch_mode().use_blink_planner().build())
> t_env.get_config().get_configuration().set_string("parallelism.default", "2")
> t_env.get_config().get_configuration().set_string(
> "python.fn-execution.bundle.size", "1")
> import datetime
> t = t_env.from_elements(
> [
> (1, 2, 3, datetime.datetime(2018, 3, 11, 3, 10, 0, 0)),
> (1, 3, 1, datetime.datetime(2018, 3, 11, 3, 10, 0, 0)),
> (1, 8, 5, datetime.datetime(2018, 3, 11, 4, 20, 0, 0))
> ],
> DataTypes.ROW(
> [DataTypes.FIELD("a", DataTypes.TINYINT()),
> DataTypes.FIELD("b", DataTypes.SMALLINT()),
> DataTypes.FIELD("c", DataTypes.INT()),
> DataTypes.FIELD("rowtime", DataTypes.TIMESTAMP(3))]))
> # sink
> t_env.execute_sql("""
> CREATE TABLE mySink (
> c INT,
> d FLOAT
> ) WITH (
> 'connector' = 'print'
> )
> """)
> t_env.create_temporary_system_function("mean_udaf", mean_udaf)
> t_env.register_function("max_add", udaf(MaxAdd(),
> result_type=DataTypes.INT(),
> func_type="pandas"))
> t_env.register_table("T", t)
> t_env.execute_sql("""
> insert into mySink
> select
> max_add(b, c)
> over (PARTITION BY a ORDER BY rowtime
> ROWS BETWEEN UNBOUNDED preceding AND 0 FOLLOWING),
> mean_udaf(b)
> over (PARTITION BY a ORDER BY rowtime
> ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING)
> from
[jira] [Updated] (FLINK-29421) Support python 3.10
[
https://issues.apache.org/jira/browse/FLINK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-29421:
-
Fix Version/s: 1.17.0
> Support python 3.10
> ---
>
> Key: FLINK-29421
> URL: https://issues.apache.org/jira/browse/FLINK-29421
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
>Reporter: Eric Sirianni
>Assignee: Huang Xingbo
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> The {{apache-flink}} package fails to install on Python 3.10 due to inability
> to compile {{numpy}}
> {noformat}
> numpy/core/src/multiarray/scalartypes.c.src:3242:12: error: too
> few arguments to function ‘_Py_HashDouble’
> 3242 | return
> _Py_HashDouble(npy_half_to_double(((PyHalfScalarObject *)obj)->obval));
> |^~
> In file included from
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/Python.h:77,
> from
> numpy/core/src/multiarray/scalartypes.c.src:3:
>
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/pyhash.h:10:23:
> note: declared here
>10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
> {noformat}
> Numpy issue https://github.com/numpy/numpy/issues/19033
> [Mailing list
> thread|https://lists.apache.org/thread/f4r9hjt1l33xf5ngnswszhnls4cxkk52]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-29421) Support python 3.10
[
https://issues.apache.org/jira/browse/FLINK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-29421:
Assignee: Huang Xingbo
> Support python 3.10
> ---
>
> Key: FLINK-29421
> URL: https://issues.apache.org/jira/browse/FLINK-29421
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
>Reporter: Eric Sirianni
>Assignee: Huang Xingbo
>Priority: Minor
> Labels: pull-request-available
>
> The {{apache-flink}} package fails to install on Python 3.10 due to inability
> to compile {{numpy}}
> {noformat}
> numpy/core/src/multiarray/scalartypes.c.src:3242:12: error: too
> few arguments to function ‘_Py_HashDouble’
> 3242 | return
> _Py_HashDouble(npy_half_to_double(((PyHalfScalarObject *)obj)->obval));
> |^~
> In file included from
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/Python.h:77,
> from
> numpy/core/src/multiarray/scalartypes.c.src:3:
>
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/pyhash.h:10:23:
> note: declared here
>10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
> {noformat}
> Numpy issue https://github.com/numpy/numpy/issues/19033
> [Mailing list
> thread|https://lists.apache.org/thread/f4r9hjt1l33xf5ngnswszhnls4cxkk52]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26974) Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656417#comment-17656417
]
Huang Xingbo commented on FLINK-26974:
--
I'm very sorry, due to several other unexpected things in last week, it may
take me a few more days to find the root cause of this unstable test. If this
unstable case affects the development progress of other developers, please ping
me at any time, and I will temporarily disable this test.
> Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
> -
>
> Key: FLINK-26974
> URL: https://issues.apache.org/jira/browse/FLINK-26974
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Mar 31 10:49:17 === FAILURES
> ===
> Mar 31 10:49:17 __
> EmbeddedThreadDependencyTests.test_add_python_file __
> Mar 31 10:49:17
> Mar 31 10:49:17 self =
> testMethod=test_add_python_file>
> Mar 31 10:49:17
> Mar 31 10:49:17 def test_add_python_file(self):
> Mar 31 10:49:17 python_file_dir = os.path.join(self.tempdir,
> "python_file_dir_" + str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir)
> Mar 31 10:49:17 python_file_path = os.path.join(python_file_dir,
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nraise
> Exception('This function should not be called!')")
> Mar 31 10:49:17 self.t_env.add_python_file(python_file_path)
> Mar 31 10:49:17
> Mar 31 10:49:17 python_file_dir_with_higher_priority = os.path.join(
> Mar 31 10:49:17 self.tempdir, "python_file_dir_" +
> str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir_with_higher_priority)
> Mar 31 10:49:17 python_file_path_higher_priority =
> os.path.join(python_file_dir_with_higher_priority,
> Mar 31 10:49:17
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path_higher_priority, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nreturn a + 2")
> Mar 31 10:49:17
> self.t_env.add_python_file(python_file_path_higher_priority)
> Mar 31 10:49:17
> Mar 31 10:49:17 def plus_two(i):
> Mar 31 10:49:17 from test_dependency_manage_lib import add_two
> Mar 31 10:49:17 return add_two(i)
> Mar 31 10:49:17
> Mar 31 10:49:17 self.t_env.create_temporary_system_function(
> Mar 31 10:49:17 "add_two", udf(plus_two, DataTypes.BIGINT(),
> DataTypes.BIGINT()))
> Mar 31 10:49:17 table_sink = source_sink_utils.TestAppendSink(
> Mar 31 10:49:17 ['a', 'b'], [DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> Mar 31 10:49:17 self.t_env.register_table_sink("Results", table_sink)
> Mar 31 10:49:17 t = self.t_env.from_elements([(1, 2), (2, 5), (3,
> 1)], ['a', 'b'])
> Mar 31 10:49:17 > t.select(expr.call("add_two", t.a),
> t.a).execute_insert("Results").wait()
> Mar 31 10:49:17
> Mar 31 10:49:17 pyflink/table/tests/test_dependency.py:63:
> Mar 31 10:49:17 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Mar 31 10:49:17 pyflink/table/table_result.py:76: in wait
> Mar 31 10:49:17 get_method(self._j_table_result, "await")()
> Mar 31 10:49:17
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34001=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=27239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29796) pyflink protobuf requirement out of date
[ https://issues.apache.org/jira/browse/FLINK-29796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-29796. Resolution: Fixed > pyflink protobuf requirement out of date > > > Key: FLINK-29796 > URL: https://issues.apache.org/jira/browse/FLINK-29796 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Jorge Villatoro >Priority: Major > > The setup.py file for pyflink currently requires protobuf<3.18 but the > dev-requirements.txt file lists protubuf<=3.21 which seems to indicate that > the library works with newer version of protobuf. The latest version of > protobuf which satisfies the requirement was 3.17.3 which was released over a > year ago, and notably the various gcloud api packages all require much newer > versions (3.19+ I think). Obviously there are ways around this but the right > answer is likely to ease/change the requirement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-28786.
Resolution: Fixed
Merged into master via e5762a558f3697294cd73da4247a741fc6f73456
Merged into release-1.16 via 4f0ffa0ddfeffbe25435595d08825cada713ac44
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.1
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29817) Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
[
https://issues.apache.org/jira/browse/FLINK-29817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-29817.
Resolution: Fixed
Merged into master via e5762a558f3697294cd73da4247a741fc6f73456
Merged into release-1.16 via d5b10d8ec9ca9fa03201ce57421bb0e714e224a7
> Published metadata for apache-flink in pypi are inconsistent and causes
> poetry to fail
> --
>
> Key: FLINK-29817
> URL: https://issues.apache.org/jira/browse/FLINK-29817
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
> Environment: macos 12.6 (M1)
> Poetry Version: 1.2.2
> Python Version: 3.9.12
>Reporter: André Casimiro
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.1
>
>
> Hi,
> Following the debug steps described in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] I got to the
> conclusion that the metadata of the apache-flink 1.16.0 package is wrong. And
> because of that I cannot properly manage my dependencies using poetry.
> I can successfully install it with pip (runs with no errors), as stated in
> the docs:
> {code:java}
> python -m pip install apache-flink {code}
> But when I try to include the dependency in my poetry project I got the
> following error:
>
> {code:java}
> ❯ poetry add apache-flink@1.16.0Updating dependencies
> Resolving dependencies... (2.0s)Because pemja (0.2.6) depends on numpy
> (1.21.4)
> and apache-flink (1.16.0) depends on numpy (>=1.14.3,<1.20), pemja (0.2.6)
> is incompatible with apache-flink (1.16.0).
> So, because cv-features depends on apache-flink (1.16.0) which depends on
> pemja (0.2.6), version solving failed. {code}
>
> I've followed the same debug steps as in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] and can confirm
> that apache-flink has exactly the same problem as described in the thread:
> the wheel package in pypi has correct dependency metadata but the pypi
> published don't.
>
> {code:java}
> ❯ pkginfo -f requires_dist
> /Users/andre/Downloads/apache_flink-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
> requires_dist: ['py4j (==0.10.9.3)', 'python-dateutil (==2.8.0)',
> 'apache-beam (==2.38.0)', 'cloudpickle (==2.1.0)', 'avro-python3
> (!=1.9.2,<1.10.0,>=1.8.1)', 'pytz (>=2018.3)', 'fastavro (<1.4.8,>=1.1.0)',
> 'requests (>=2.26.0)', 'protobuf (<3.18)', 'httplib2 (<=0.20.4,>=0.19.0)',
> 'apache-flink-libraries (<1.16.1,>=1.16.0)', 'numpy (<1.22.0,>=1.21.4)',
> 'pandas (<1.4.0,>=1.3.0)', 'pyarrow (<9.0.0,>=5.0.0)', 'pemja (==0.2.6) ;
> python_full_version >= "3.7" and platform_system != "Windows"'] {code}
> but the pipy json metadata is wrong:
>
>
> {code:java}
> ❯ curl -sL https://pypi.org/pypi/apache-flink/json | jq '.info.requires_dist'[
> "py4j (==0.10.9.3)",
> "python-dateutil (==2.8.0)",
> "apache-beam (==2.38.0)",
> "cloudpickle (==2.1.0)",
> "avro-python3 (!=1.9.2,<1.10.0,>=1.8.1)",
> "pytz (>=2018.3)",
> "fastavro (<1.4.8,>=1.1.0)",
> "requests (>=2.26.0)",
> "protobuf (<3.18)",
> "httplib2 (<=0.20.4,>=0.19.0)",
> "apache-flink-libraries (<1.16.1,>=1.16.0)",
> "numpy (<1.20,>=1.14.3)",
> "pandas (<1.2.0,>=1.0)",
> "pyarrow (<7.0.0,>=0.15.1)",
> "pemja (==0.2.6) ; python_full_version >= \"3.7\" and platform_system !=
> \"Windows\""
> ]{code}
>
> As per [this
> comment|https://github.com/python-poetry/poetry/issues/3011#issuecomment-702826616],
> could you please republish the package correcting this metadata information,
> please? This [other
> comment|https://github.com/apple/turicreate/issues/3342#issuecomment-702957550]
> can help gain more context.
>
> Thanks
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30040) Add more PyFlink API examples
Huang Xingbo created FLINK-30040: Summary: Add more PyFlink API examples Key: FLINK-30040 URL: https://issues.apache.org/jira/browse/FLINK-30040 Project: Flink Issue Type: Sub-task Components: API / Python, Documentation Affects Versions: 1.16.0, 1.15.3 Reporter: Huang Xingbo Many APIs of PyFlink currently do not provide corresponding examples -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-21223) Support to specify the input/output types of Python UDFs via string
[
https://issues.apache.org/jira/browse/FLINK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-21223:
Assignee: Huang Xingbo
> Support to specify the input/output types of Python UDFs via string
> ---
>
> Key: FLINK-21223
> URL: https://issues.apache.org/jira/browse/FLINK-21223
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Huang Xingbo
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> Currently, users need to specify the input/output types as following:
> {code}
> {{@udf(result_type=DataTypes.BIGINT())
> def add(i, j):
> return i + j
> }}{code}
> [FLIP-65|https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs]
> makes it possible to support syntaxes as following:
> {code}
> {{@udf(result_type="BIGINT")
> def add(i, j):
> return i + j
> }}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29966) Replace and redesign the Python api documentation base
[ https://issues.apache.org/jira/browse/FLINK-29966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-29966. Fix Version/s: 1.16.1 1.15.4 Resolution: Fixed Merged into master via b838a4e76e226a0567d39db1f9c29305cb6d913f Merged into release-1.16 via 7d5585dd9ea4f26bf28c43cbb5207f2a5b74420b Merged into release-1.15 via d6bf30a013569c4a8e0cd546f3105296c7b32efe > Replace and redesign the Python api documentation base > -- > > Key: FLINK-29966 > URL: https://issues.apache.org/jira/browse/FLINK-29966 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Affects Versions: 1.16.0, 1.15.3 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.1, 1.15.4 > > > The doc of the existing python api is difficult to read and use. I have a > demo site for redesigning the Python api documentation base. See > https://pyflink-api-docs-test.readthedocs.io/en/latest/ as an example. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28394) Python py36-cython: InvocationError for command install_command.sh fails with exit code 1
[
https://issues.apache.org/jira/browse/FLINK-28394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634177#comment-17634177
]
Huang Xingbo commented on FLINK-28394:
--
`Python py36-cython` is not the cause of the actual error. Any test running
under Python 3.6 fails, and then `Python py36-cython` will be displayed at the
end. You need to scroll up in the log to find out the actual failed test. I
looked at the links posted and they are all about
https://issues.apache.org/jira/browse/FLINK-29461.
> Python py36-cython: InvocationError for command install_command.sh fails with
> exit code 1
> -
>
> Key: FLINK-28394
> URL: https://issues.apache.org/jira/browse/FLINK-28394
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0, 1.15.3
>Reporter: Martijn Visser
>Assignee: Huang Xingbo
>Priority: Major
> Labels: stale-assigned, test-stability
>
> {code:java}
> Jul 05 03:47:22 Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> Jul 05 03:47:32 Using Python version 3.8.13 (default, Mar 28 2022 11:38:47)
> Jul 05 03:47:32 pip_test_code.py success!
> Jul 05 03:47:32 py38-cython finish: run-test after 1658.14 seconds
> Jul 05 03:47:32 py38-cython start: run-test-post
> Jul 05 03:47:32 py38-cython finish: run-test-post after 0.00 seconds
> Jul 05 03:47:32 ___ summary
>
> Jul 05 03:47:32 ERROR: py36-cython: InvocationError for command
> /__w/3/s/flink-python/dev/install_command.sh --exists-action w
> .tox/.tmp/package/1/apache-flink-1.15.dev0.zip (exited with code 1)
> Jul 05 03:47:32 py37-cython: commands succeeded
> Jul 05 03:47:32 py38-cython: commands succeeded
> Jul 05 03:47:32 cleanup
> /__w/3/s/flink-python/.tox/.tmp/package/1/apache-flink-1.15.dev0.zip
> Jul 05 03:47:33 tox checks... [FAILED]
> Jul 05 03:47:33 Process exited with EXIT CODE: 1.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37604=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=85189c57-d8a0-5c9c-b61d-fc05cfac62cf=27789
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-28957) Optimize Python API doc
[ https://issues.apache.org/jira/browse/FLINK-28957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo reassigned FLINK-28957: Assignee: Huang Xingbo > Optimize Python API doc > --- > > Key: FLINK-28957 > URL: https://issues.apache.org/jira/browse/FLINK-28957 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Affects Versions: 1.14.5, 1.15.1, 1.16.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > > The overall style and structure of the current python api doc is not so > convenient to new users to learn and use, we can use numpydoc or other > document styles to optimize. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29966) Replace and redesign the Python api documentation base
Huang Xingbo created FLINK-29966: Summary: Replace and redesign the Python api documentation base Key: FLINK-29966 URL: https://issues.apache.org/jira/browse/FLINK-29966 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Affects Versions: 1.15.2, 1.16.0 Reporter: Huang Xingbo Assignee: Huang Xingbo The doc of the existing python api is difficult to read and use. I have a demo site for redesigning the Python api documentation base. See https://pyflink-api-docs-test.readthedocs.io/en/latest/ as an example. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-28786:
-
Fix Version/s: 1.17.0
1.16.1
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Fix For: 1.17.0, 1.16.1
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reopened FLINK-28786:
--
Assignee: Huang Xingbo
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-26974) Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-26974:
Assignee: Huang Xingbo
> Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
> -
>
> Key: FLINK-26974
> URL: https://issues.apache.org/jira/browse/FLINK-26974
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Mar 31 10:49:17 === FAILURES
> ===
> Mar 31 10:49:17 __
> EmbeddedThreadDependencyTests.test_add_python_file __
> Mar 31 10:49:17
> Mar 31 10:49:17 self =
> testMethod=test_add_python_file>
> Mar 31 10:49:17
> Mar 31 10:49:17 def test_add_python_file(self):
> Mar 31 10:49:17 python_file_dir = os.path.join(self.tempdir,
> "python_file_dir_" + str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir)
> Mar 31 10:49:17 python_file_path = os.path.join(python_file_dir,
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nraise
> Exception('This function should not be called!')")
> Mar 31 10:49:17 self.t_env.add_python_file(python_file_path)
> Mar 31 10:49:17
> Mar 31 10:49:17 python_file_dir_with_higher_priority = os.path.join(
> Mar 31 10:49:17 self.tempdir, "python_file_dir_" +
> str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir_with_higher_priority)
> Mar 31 10:49:17 python_file_path_higher_priority =
> os.path.join(python_file_dir_with_higher_priority,
> Mar 31 10:49:17
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path_higher_priority, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nreturn a + 2")
> Mar 31 10:49:17
> self.t_env.add_python_file(python_file_path_higher_priority)
> Mar 31 10:49:17
> Mar 31 10:49:17 def plus_two(i):
> Mar 31 10:49:17 from test_dependency_manage_lib import add_two
> Mar 31 10:49:17 return add_two(i)
> Mar 31 10:49:17
> Mar 31 10:49:17 self.t_env.create_temporary_system_function(
> Mar 31 10:49:17 "add_two", udf(plus_two, DataTypes.BIGINT(),
> DataTypes.BIGINT()))
> Mar 31 10:49:17 table_sink = source_sink_utils.TestAppendSink(
> Mar 31 10:49:17 ['a', 'b'], [DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> Mar 31 10:49:17 self.t_env.register_table_sink("Results", table_sink)
> Mar 31 10:49:17 t = self.t_env.from_elements([(1, 2), (2, 5), (3,
> 1)], ['a', 'b'])
> Mar 31 10:49:17 > t.select(expr.call("add_two", t.a),
> t.a).execute_insert("Results").wait()
> Mar 31 10:49:17
> Mar 31 10:49:17 pyflink/table/tests/test_dependency.py:63:
> Mar 31 10:49:17 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Mar 31 10:49:17 pyflink/table/table_result.py:76: in wait
> Mar 31 10:49:17 get_method(self._j_table_result, "await")()
> Mar 31 10:49:17
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34001=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=27239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-29461) ProcessDataStreamStreamingTests.test_process_function unstable
[
https://issues.apache.org/jira/browse/FLINK-29461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-29461:
Assignee: Huang Xingbo
> ProcessDataStreamStreamingTests.test_process_function unstable
> --
>
> Key: FLINK-29461
> URL: https://issues.apache.org/jira/browse/FLINK-29461
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-29T02:10:45.3571648Z Sep 29 02:10:45 self =
> testMethod=test_process_function>
> 2022-09-29T02:10:45.3572279Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3572810Z Sep 29 02:10:45 def
> test_process_function(self):
> 2022-09-29T02:10:45.3573495Z Sep 29 02:10:45
> self.env.set_parallelism(1)
> 2022-09-29T02:10:45.3574148Z Sep 29 02:10:45
> self.env.get_config().set_auto_watermark_interval(2000)
> 2022-09-29T02:10:45.3580634Z Sep 29 02:10:45
> self.env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
> 2022-09-29T02:10:45.3583194Z Sep 29 02:10:45 data_stream =
> self.env.from_collection([(1, '1603708211000'),
> 2022-09-29T02:10:45.3584515Z Sep 29 02:10:45
>(2, '1603708224000'),
> 2022-09-29T02:10:45.3585957Z Sep 29 02:10:45
>(3, '1603708226000'),
> 2022-09-29T02:10:45.3587132Z Sep 29 02:10:45
>(4, '1603708289000')],
> 2022-09-29T02:10:45.3588094Z Sep 29 02:10:45
> type_info=Types.ROW([Types.INT(), Types.STRING()]))
> 2022-09-29T02:10:45.3589090Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3589949Z Sep 29 02:10:45 class
> MyProcessFunction(ProcessFunction):
> 2022-09-29T02:10:45.3590710Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3591856Z Sep 29 02:10:45 def
> process_element(self, value, ctx):
> 2022-09-29T02:10:45.3592873Z Sep 29 02:10:45
> current_timestamp = ctx.timestamp()
> 2022-09-29T02:10:45.3593862Z Sep 29 02:10:45
> current_watermark = ctx.timer_service().current_watermark()
> 2022-09-29T02:10:45.3594915Z Sep 29 02:10:45 yield "current
> timestamp: {}, current watermark: {}, current_value: {}"\
> 2022-09-29T02:10:45.3596201Z Sep 29 02:10:45
> .format(str(current_timestamp), str(current_watermark), str(value))
> 2022-09-29T02:10:45.3597089Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3597942Z Sep 29 02:10:45 watermark_strategy =
> WatermarkStrategy.for_monotonous_timestamps()\
> 2022-09-29T02:10:45.3599260Z Sep 29 02:10:45
> .with_timestamp_assigner(SecondColumnTimestampAssigner())
> 2022-09-29T02:10:45.3600611Z Sep 29 02:10:45
> data_stream.assign_timestamps_and_watermarks(watermark_strategy)\
> 2022-09-29T02:10:45.3601877Z Sep 29 02:10:45
> .process(MyProcessFunction(),
> output_type=Types.STRING()).add_sink(self.test_sink)
> 2022-09-29T02:10:45.3603527Z Sep 29 02:10:45 self.env.execute('test
> process function')
> 2022-09-29T02:10:45.3604445Z Sep 29 02:10:45 results =
> self.test_sink.get_results()
> 2022-09-29T02:10:45.3605684Z Sep 29 02:10:45 expected = ["current
> timestamp: 1603708211000, current watermark: "
> 2022-09-29T02:10:45.3607157Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=1, f1='1603708211000')",
> 2022-09-29T02:10:45.3608256Z Sep 29 02:10:45 "current
> timestamp: 1603708224000, current watermark: "
> 2022-09-29T02:10:45.3609650Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=2, f1='1603708224000')",
> 2022-09-29T02:10:45.3610854Z Sep 29 02:10:45 "current
> timestamp: 1603708226000, current watermark: "
> 2022-09-29T02:10:45.3612279Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=3, f1='1603708226000')",
> 2022-09-29T02:10:45.3613382Z Sep 29 02:10:45 "current
> timestamp: 1603708289000, current watermark: "
> 2022-09-29T02:10:45.3615683Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=4, f1='1603708289000')"]
> 2022-09-29T02:10:45.3617687Z Sep 29 02:10:45 >
> self.assert_equals_sorted(expected, results)
> 2022-09-29T02:10:45.3618620Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3619425Z Sep 29 02:10:45
> pyflink/datastream/tests/test_data_stream.py:986:
> 2022-09-29T02:10:45.3620424Z Sep 29 02:10:45 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2022-09-29T02:10:45.3621886Z Sep
[jira] [Assigned] (FLINK-29817) Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
[
https://issues.apache.org/jira/browse/FLINK-29817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-29817:
Assignee: Huang Xingbo
> Published metadata for apache-flink in pypi are inconsistent and causes
> poetry to fail
> --
>
> Key: FLINK-29817
> URL: https://issues.apache.org/jira/browse/FLINK-29817
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
> Environment: macos 12.6 (M1)
> Poetry Version: 1.2.2
> Python Version: 3.9.12
>Reporter: André Casimiro
>Assignee: Huang Xingbo
>Priority: Critical
>
> Hi,
> Following the debug steps described in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] I got to the
> conclusion that the metadata of the apache-flink 1.16.0 package is wrong. And
> because of that I cannot properly manage my dependencies using poetry.
> I can successfully install it with pip (runs with no errors), as stated in
> the docs:
> {code:java}
> python -m pip install apache-flink {code}
> But when I try to include the dependency in my poetry project I got the
> following error:
>
> {code:java}
> ❯ poetry add apache-flink@1.16.0Updating dependencies
> Resolving dependencies... (2.0s)Because pemja (0.2.6) depends on numpy
> (1.21.4)
> and apache-flink (1.16.0) depends on numpy (>=1.14.3,<1.20), pemja (0.2.6)
> is incompatible with apache-flink (1.16.0).
> So, because cv-features depends on apache-flink (1.16.0) which depends on
> pemja (0.2.6), version solving failed. {code}
>
> I've followed the same debug steps as in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] and can confirm
> that apache-flink has exactly the same problem as described in the thread:
> the wheel package in pypi has correct dependency metadata but the pypi
> published don't.
>
> {code:java}
> ❯ pkginfo -f requires_dist
> /Users/andre/Downloads/apache_flink-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
> requires_dist: ['py4j (==0.10.9.3)', 'python-dateutil (==2.8.0)',
> 'apache-beam (==2.38.0)', 'cloudpickle (==2.1.0)', 'avro-python3
> (!=1.9.2,<1.10.0,>=1.8.1)', 'pytz (>=2018.3)', 'fastavro (<1.4.8,>=1.1.0)',
> 'requests (>=2.26.0)', 'protobuf (<3.18)', 'httplib2 (<=0.20.4,>=0.19.0)',
> 'apache-flink-libraries (<1.16.1,>=1.16.0)', 'numpy (<1.22.0,>=1.21.4)',
> 'pandas (<1.4.0,>=1.3.0)', 'pyarrow (<9.0.0,>=5.0.0)', 'pemja (==0.2.6) ;
> python_full_version >= "3.7" and platform_system != "Windows"'] {code}
> but the pipy json metadata is wrong:
>
>
> {code:java}
> ❯ curl -sL https://pypi.org/pypi/apache-flink/json | jq '.info.requires_dist'[
> "py4j (==0.10.9.3)",
> "python-dateutil (==2.8.0)",
> "apache-beam (==2.38.0)",
> "cloudpickle (==2.1.0)",
> "avro-python3 (!=1.9.2,<1.10.0,>=1.8.1)",
> "pytz (>=2018.3)",
> "fastavro (<1.4.8,>=1.1.0)",
> "requests (>=2.26.0)",
> "protobuf (<3.18)",
> "httplib2 (<=0.20.4,>=0.19.0)",
> "apache-flink-libraries (<1.16.1,>=1.16.0)",
> "numpy (<1.20,>=1.14.3)",
> "pandas (<1.2.0,>=1.0)",
> "pyarrow (<7.0.0,>=0.15.1)",
> "pemja (==0.2.6) ; python_full_version >= \"3.7\" and platform_system !=
> \"Windows\""
> ]{code}
>
> As per [this
> comment|https://github.com/python-poetry/poetry/issues/3011#issuecomment-702826616],
> could you please republish the package correcting this metadata information,
> please? This [other
> comment|https://github.com/apple/turicreate/issues/3342#issuecomment-702957550]
> can help gain more context.
>
> Thanks
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29817) Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
[
https://issues.apache.org/jira/browse/FLINK-29817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-29817:
-
Fix Version/s: 1.17.0
1.16.1
> Published metadata for apache-flink in pypi are inconsistent and causes
> poetry to fail
> --
>
> Key: FLINK-29817
> URL: https://issues.apache.org/jira/browse/FLINK-29817
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
> Environment: macos 12.6 (M1)
> Poetry Version: 1.2.2
> Python Version: 3.9.12
>Reporter: André Casimiro
>Assignee: Huang Xingbo
>Priority: Critical
> Fix For: 1.17.0, 1.16.1
>
>
> Hi,
> Following the debug steps described in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] I got to the
> conclusion that the metadata of the apache-flink 1.16.0 package is wrong. And
> because of that I cannot properly manage my dependencies using poetry.
> I can successfully install it with pip (runs with no errors), as stated in
> the docs:
> {code:java}
> python -m pip install apache-flink {code}
> But when I try to include the dependency in my poetry project I got the
> following error:
>
> {code:java}
> ❯ poetry add apache-flink@1.16.0Updating dependencies
> Resolving dependencies... (2.0s)Because pemja (0.2.6) depends on numpy
> (1.21.4)
> and apache-flink (1.16.0) depends on numpy (>=1.14.3,<1.20), pemja (0.2.6)
> is incompatible with apache-flink (1.16.0).
> So, because cv-features depends on apache-flink (1.16.0) which depends on
> pemja (0.2.6), version solving failed. {code}
>
> I've followed the same debug steps as in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] and can confirm
> that apache-flink has exactly the same problem as described in the thread:
> the wheel package in pypi has correct dependency metadata but the pypi
> published don't.
>
> {code:java}
> ❯ pkginfo -f requires_dist
> /Users/andre/Downloads/apache_flink-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
> requires_dist: ['py4j (==0.10.9.3)', 'python-dateutil (==2.8.0)',
> 'apache-beam (==2.38.0)', 'cloudpickle (==2.1.0)', 'avro-python3
> (!=1.9.2,<1.10.0,>=1.8.1)', 'pytz (>=2018.3)', 'fastavro (<1.4.8,>=1.1.0)',
> 'requests (>=2.26.0)', 'protobuf (<3.18)', 'httplib2 (<=0.20.4,>=0.19.0)',
> 'apache-flink-libraries (<1.16.1,>=1.16.0)', 'numpy (<1.22.0,>=1.21.4)',
> 'pandas (<1.4.0,>=1.3.0)', 'pyarrow (<9.0.0,>=5.0.0)', 'pemja (==0.2.6) ;
> python_full_version >= "3.7" and platform_system != "Windows"'] {code}
> but the pipy json metadata is wrong:
>
>
> {code:java}
> ❯ curl -sL https://pypi.org/pypi/apache-flink/json | jq '.info.requires_dist'[
> "py4j (==0.10.9.3)",
> "python-dateutil (==2.8.0)",
> "apache-beam (==2.38.0)",
> "cloudpickle (==2.1.0)",
> "avro-python3 (!=1.9.2,<1.10.0,>=1.8.1)",
> "pytz (>=2018.3)",
> "fastavro (<1.4.8,>=1.1.0)",
> "requests (>=2.26.0)",
> "protobuf (<3.18)",
> "httplib2 (<=0.20.4,>=0.19.0)",
> "apache-flink-libraries (<1.16.1,>=1.16.0)",
> "numpy (<1.20,>=1.14.3)",
> "pandas (<1.2.0,>=1.0)",
> "pyarrow (<7.0.0,>=0.15.1)",
> "pemja (==0.2.6) ; python_full_version >= \"3.7\" and platform_system !=
> \"Windows\""
> ]{code}
>
> As per [this
> comment|https://github.com/python-poetry/poetry/issues/3011#issuecomment-702826616],
> could you please republish the package correcting this metadata information,
> please? This [other
> comment|https://github.com/apple/turicreate/issues/3342#issuecomment-702957550]
> can help gain more context.
>
> Thanks
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29817) Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
[
https://issues.apache.org/jira/browse/FLINK-29817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627413#comment-17627413
]
Huang Xingbo commented on FLINK-29817:
--
I think we can optimize the `metadata` to solve this problem in the 1.16.1.
> Published metadata for apache-flink in pypi are inconsistent and causes
> poetry to fail
> --
>
> Key: FLINK-29817
> URL: https://issues.apache.org/jira/browse/FLINK-29817
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
> Environment: macos 12.6 (M1)
> Poetry Version: 1.2.2
> Python Version: 3.9.12
>Reporter: André Casimiro
>Priority: Critical
>
> Hi,
> Following the debug steps described in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] I got to the
> conclusion that the metadata of the apache-flink 1.16.0 package is wrong. And
> because of that I cannot properly manage my dependencies using poetry.
> I can successfully install it with pip (runs with no errors), as stated in
> the docs:
> {code:java}
> python -m pip install apache-flink {code}
> But when I try to include the dependency in my poetry project I got the
> following error:
>
> {code:java}
> ❯ poetry add apache-flink@1.16.0Updating dependencies
> Resolving dependencies... (2.0s)Because pemja (0.2.6) depends on numpy
> (1.21.4)
> and apache-flink (1.16.0) depends on numpy (>=1.14.3,<1.20), pemja (0.2.6)
> is incompatible with apache-flink (1.16.0).
> So, because cv-features depends on apache-flink (1.16.0) which depends on
> pemja (0.2.6), version solving failed. {code}
>
> I've followed the same debug steps as in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] and can confirm
> that apache-flink has exactly the same problem as described in the thread:
> the wheel package in pypi has correct dependency metadata but the pypi
> published don't.
>
> {code:java}
> ❯ pkginfo -f requires_dist
> /Users/andre/Downloads/apache_flink-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
> requires_dist: ['py4j (==0.10.9.3)', 'python-dateutil (==2.8.0)',
> 'apache-beam (==2.38.0)', 'cloudpickle (==2.1.0)', 'avro-python3
> (!=1.9.2,<1.10.0,>=1.8.1)', 'pytz (>=2018.3)', 'fastavro (<1.4.8,>=1.1.0)',
> 'requests (>=2.26.0)', 'protobuf (<3.18)', 'httplib2 (<=0.20.4,>=0.19.0)',
> 'apache-flink-libraries (<1.16.1,>=1.16.0)', 'numpy (<1.22.0,>=1.21.4)',
> 'pandas (<1.4.0,>=1.3.0)', 'pyarrow (<9.0.0,>=5.0.0)', 'pemja (==0.2.6) ;
> python_full_version >= "3.7" and platform_system != "Windows"'] {code}
> but the pipy json metadata is wrong:
>
>
> {code:java}
> ❯ curl -sL https://pypi.org/pypi/apache-flink/json | jq '.info.requires_dist'[
> "py4j (==0.10.9.3)",
> "python-dateutil (==2.8.0)",
> "apache-beam (==2.38.0)",
> "cloudpickle (==2.1.0)",
> "avro-python3 (!=1.9.2,<1.10.0,>=1.8.1)",
> "pytz (>=2018.3)",
> "fastavro (<1.4.8,>=1.1.0)",
> "requests (>=2.26.0)",
> "protobuf (<3.18)",
> "httplib2 (<=0.20.4,>=0.19.0)",
> "apache-flink-libraries (<1.16.1,>=1.16.0)",
> "numpy (<1.20,>=1.14.3)",
> "pandas (<1.2.0,>=1.0)",
> "pyarrow (<7.0.0,>=0.15.1)",
> "pemja (==0.2.6) ; python_full_version >= \"3.7\" and platform_system !=
> \"Windows\""
> ]{code}
>
> As per [this
> comment|https://github.com/python-poetry/poetry/issues/3011#issuecomment-702826616],
> could you please republish the package correcting this metadata information,
> please? This [other
> comment|https://github.com/apple/turicreate/issues/3342#issuecomment-702957550]
> can help gain more context.
>
> Thanks
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29833) Improve PyFlink support in Windows
Huang Xingbo created FLINK-29833: Summary: Improve PyFlink support in Windows Key: FLINK-29833 URL: https://issues.apache.org/jira/browse/FLINK-29833 Project: Flink Issue Type: Improvement Components: API / Python Affects Versions: 1.15.2, 1.16.0 Reporter: Huang Xingbo Assignee: Huang Xingbo Many users are used to developing PyFlink jobs on Windows. It is necessary to improve the simplicity of PyFlink job development on Windows -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29567) Revert sink output metric names from numRecordsSend back to numRecordsOut
[
https://issues.apache.org/jira/browse/FLINK-29567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626927#comment-17626927
]
Huang Xingbo commented on FLINK-29567:
--
Hi [~renqs], the fixed PR hasn't been backported to release-1.15?
> Revert sink output metric names from numRecordsSend back to numRecordsOut
> -
>
> Key: FLINK-29567
> URL: https://issues.apache.org/jira/browse/FLINK-29567
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.16.0, 1.15.3
>Reporter: Qingsheng Ren
>Assignee: Qingsheng Ren
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.16.0, 1.15.3
>
>
> As discussed in [the mailing
> list|https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv],
> all{color:#33} sink metrics with name “numXXXOut” defined in FLIP-33 are
> replace by “numXXXSend” in FLINK-26126 and FLINK-26492. {color}
> {color:#33}Considering metric names are public APIs, this is a breaking
> change to end users and not backward compatible. We need to revert these
> metric names back. {color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29817) Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
[
https://issues.apache.org/jira/browse/FLINK-29817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626911#comment-17626911
]
Huang Xingbo commented on FLINK-29817:
--
The `install_requirements` of PyFlink depends on your `python_version`.
{code:python}
if sys.version_info < (3, 7):
# python 3.6 upper and lower limit
install_requires.append('numpy>=1.14.3,<1.20')
install_requires.append('pandas>=1.0,<1.2.0')
install_requires.append('pyarrow>=0.15.1,<7.0.0')
else:
# python 3.7, 3.8 and 3.9 upper limit and M1 chip lower limit,
install_requires.append('numpy>=1.21.4,<1.22.0')
install_requires.append('pandas>=1.3.0,<1.4.0')
install_requires.append('pyarrow>=5.0.0,<9.0.0')
{code}
So I guess Pypi collects the `metadata` used in Python3.6.
> Published metadata for apache-flink in pypi are inconsistent and causes
> poetry to fail
> --
>
> Key: FLINK-29817
> URL: https://issues.apache.org/jira/browse/FLINK-29817
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
> Environment: macos 12.6 (M1)
> Poetry Version: 1.2.2
> Python Version: 3.9.12
>Reporter: André Casimiro
>Priority: Critical
>
> Hi,
> Following the debug steps described in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] I got to the
> conclusion that the metadata of the apache-flink 1.16.0 package is wrong. And
> because of that I cannot properly manage my dependencies using poetry.
> I can successfully install it with pip (runs with no errors), as stated in
> the docs:
> {code:java}
> python -m pip install apache-flink {code}
> But when I try to include the dependency in my poetry project I got the
> following error:
>
> {code:java}
> ❯ poetry add apache-flink@1.16.0Updating dependencies
> Resolving dependencies... (2.0s)Because pemja (0.2.6) depends on numpy
> (1.21.4)
> and apache-flink (1.16.0) depends on numpy (>=1.14.3,<1.20), pemja (0.2.6)
> is incompatible with apache-flink (1.16.0).
> So, because cv-features depends on apache-flink (1.16.0) which depends on
> pemja (0.2.6), version solving failed. {code}
>
> I've followed the same debug steps as in [this github
> thread|https://github.com/python-poetry/poetry/issues/3011] and can confirm
> that apache-flink has exactly the same problem as described in the thread:
> the wheel package in pypi has correct dependency metadata but the pypi
> published don't.
>
> {code:java}
> ❯ pkginfo -f requires_dist
> /Users/andre/Downloads/apache_flink-1.16.0-cp39-cp39-macosx_11_0_arm64.whl
> requires_dist: ['py4j (==0.10.9.3)', 'python-dateutil (==2.8.0)',
> 'apache-beam (==2.38.0)', 'cloudpickle (==2.1.0)', 'avro-python3
> (!=1.9.2,<1.10.0,>=1.8.1)', 'pytz (>=2018.3)', 'fastavro (<1.4.8,>=1.1.0)',
> 'requests (>=2.26.0)', 'protobuf (<3.18)', 'httplib2 (<=0.20.4,>=0.19.0)',
> 'apache-flink-libraries (<1.16.1,>=1.16.0)', 'numpy (<1.22.0,>=1.21.4)',
> 'pandas (<1.4.0,>=1.3.0)', 'pyarrow (<9.0.0,>=5.0.0)', 'pemja (==0.2.6) ;
> python_full_version >= "3.7" and platform_system != "Windows"'] {code}
> but the pipy json metadata is wrong:
>
>
> {code:java}
> ❯ curl -sL https://pypi.org/pypi/apache-flink/json | jq '.info.requires_dist'[
> "py4j (==0.10.9.3)",
> "python-dateutil (==2.8.0)",
> "apache-beam (==2.38.0)",
> "cloudpickle (==2.1.0)",
> "avro-python3 (!=1.9.2,<1.10.0,>=1.8.1)",
> "pytz (>=2018.3)",
> "fastavro (<1.4.8,>=1.1.0)",
> "requests (>=2.26.0)",
> "protobuf (<3.18)",
> "httplib2 (<=0.20.4,>=0.19.0)",
> "apache-flink-libraries (<1.16.1,>=1.16.0)",
> "numpy (<1.20,>=1.14.3)",
> "pandas (<1.2.0,>=1.0)",
> "pyarrow (<7.0.0,>=0.15.1)",
> "pemja (==0.2.6) ; python_full_version >= \"3.7\" and platform_system !=
> \"Windows\""
> ]{code}
>
> As per [this
> comment|https://github.com/python-poetry/poetry/issues/3011#issuecomment-702826616],
> could you please republish the package correcting this metadata information,
> please? This [other
> comment|https://github.com/apple/turicreate/issues/3342#issuecomment-702957550]
> can help gain more context.
>
> Thanks
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29733) Error Flink connector hive Test failing
[ https://issues.apache.org/jira/browse/FLINK-29733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-29733. Fix Version/s: 1.17.0 Assignee: Samrat Deb Resolution: Fixed Merged into master via 759d954960cbec28437955e5f482788902342035 > Error Flink connector hive Test failing > --- > > Key: FLINK-29733 > URL: https://issues.apache.org/jira/browse/FLINK-29733 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.17.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42328=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=d04c9862-880c-52f5-574b-a7a79fef8e0f] > This is caused by FLINK-29478 > reported by [~hxbks2ks] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-29788. Fix Version/s: 1.17.0 1.16.1 Assignee: Hangxiang Yu Resolution: Fixed Merged into master via c89e400ae379c8b7490d9af20f82f49319895dce Merged into release-1.16 via 87d86c05c7a91d7546935ec1ceee3c0c55e2e191 > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Assignee: Hangxiang Yu >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0, 1.16.1 > > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626434#comment-17626434 ] Huang Xingbo commented on FLINK-29788: -- [~masteryhx] Thanks a lot for the fix. Before the release, the code may change, so I think preparing the migrated test data in advance may result in some problems. > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625491#comment-17625491 ] Huang Xingbo commented on FLINK-29788: -- It is not the blocker of release 1.16.0，but it belongs to the finalized step of a release. > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Critical > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29788: - Priority: Critical (was: Major) > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Critical > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625404#comment-17625404 ] Huang Xingbo commented on FLINK-29788: -- [~masteryhx] Could you help take a look? Thx. > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Major > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
Huang Xingbo created FLINK-29788: Summary: StatefulJobWBroadcastStateMigrationITCase failed in native savepoints Key: FLINK-29788 URL: https://issues.apache.org/jira/browse/FLINK-29788 Project: Flink Issue Type: Bug Components: Release System, Runtime / State Backends Affects Versions: 1.16.0 Reporter: Huang Xingbo Attachments: image-2022-10-28-11-18-45-471.png !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611629#comment-17611629
]
Huang Xingbo commented on FLINK-29427:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41488=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=21930
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:20.9521677Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3927)
>
[jira] [Commented] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611628#comment-17611628
]
Huang Xingbo commented on FLINK-29427:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41493=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=21545
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:20.9521677Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3927)
>
[jira] [Commented] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611627#comment-17611627
]
Huang Xingbo commented on FLINK-29419:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41494=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=11782
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Weijie Guo
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29483) flink python udf arrow in thread model bug
[ https://issues.apache.org/jira/browse/FLINK-29483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611617#comment-17611617 ] Huang Xingbo commented on FLINK-29483: -- [~jackylau] You means your job runs in `thread mode` will raise this exception? > flink python udf arrow in thread model bug > -- > > Key: FLINK-29483 > URL: https://issues.apache.org/jira/browse/FLINK-29483 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.16.0, 1.15.2 >Reporter: jackylau >Priority: Major > Fix For: 1.17.0, 1.15.3, 1.16.1 > > Attachments: image-2022-09-30-17-03-05-005.png > > > !image-2022-09-30-17-03-05-005.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29479) support whether using system env for pythonpath
[ https://issues.apache.org/jira/browse/FLINK-29479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611469#comment-17611469 ] Huang Xingbo commented on FLINK-29479: -- [~jackylau] From my side, the users dependency will have a higher priority, so it shouldn't be affected by the PYTHONPATH of the system env under normal circumstances. Could you give a more specific example of how the system's PYTHONPATH is having a bad effect in some circumstance. > support whether using system env for pythonpath > --- > > Key: FLINK-29479 > URL: https://issues.apache.org/jira/browse/FLINK-29479 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.17.0 >Reporter: jackylau >Priority: Major > Fix For: 1.17.0 > > > It exists PYTHONPATH env in system,like yarn/k8s images, it will cause > conflict with users python depdendency sometimes. so i suggest add a config > to do whether using system env of PYTHONPATH -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29350) Add a section for moving planner jar in Hive dependencies page
[ https://issues.apache.org/jira/browse/FLINK-29350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29350: - Fix Version/s: 1.17.0 (was: 1.16.0) > Add a section for moving planner jar in Hive dependencies page > -- > > Key: FLINK-29350 > URL: https://issues.apache.org/jira/browse/FLINK-29350 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Affects Versions: 1.16.0 >Reporter: luoyuxia >Assignee: luoyuxia >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28071) Support missing built-in functions in Table API
[ https://issues.apache.org/jira/browse/FLINK-28071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-28071: - Fix Version/s: 1.17.0 (was: 1.16.0) > Support missing built-in functions in Table API > --- > > Key: FLINK-28071 > URL: https://issues.apache.org/jira/browse/FLINK-28071 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Dian Fu >Assignee: Luning Wang >Priority: Major > Fix For: 1.17.0 > > > There are many built-in functions are not supported. See > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/ > for more details. There are two columns for each built-in function: *SQL > Function* and *Table Function*, if a function is not supported in *Table > API*, the *Table Function* column is documented as *N/A*. We need to evaluate > each of these functions to ensure that they could be used in both SQL and > Table API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27467) Remove CliFrontendTestBase
[ https://issues.apache.org/jira/browse/FLINK-27467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27467: - Fix Version/s: 1.17.0 (was: 1.16.0) > Remove CliFrontendTestBase > -- > > Key: FLINK-27467 > URL: https://issues.apache.org/jira/browse/FLINK-27467 > Project: Flink > Issue Type: Technical Debt > Components: Command Line Client, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > > The class isn't really providing any value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29372) Add a suffix to keys that violate YAML spec
[
https://issues.apache.org/jira/browse/FLINK-29372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-29372:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Add a suffix to keys that violate YAML spec
> ---
>
> Key: FLINK-29372
> URL: https://issues.apache.org/jira/browse/FLINK-29372
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Configuration
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> We have a few options where the key is a prefix of other options (e.g.,
> {{high-availability}} and {{high-availability.cluster-id}}.
> Add a suffix to these options and keep the old key as deprecated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29430) Sanity check in InternalKeyContextImpl#setCurrentKeyGroupIndex
[
https://issues.apache.org/jira/browse/FLINK-29430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-29430:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Sanity check in InternalKeyContextImpl#setCurrentKeyGroupIndex
> --
>
> Key: FLINK-29430
> URL: https://issues.apache.org/jira/browse/FLINK-29430
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.15.2
>Reporter: Zakelly Lan
>Assignee: Zakelly Lan
>Priority: Major
> Fix For: 1.17.0
>
>
> Currently the HeapStateBackend check whether the current key group index is a
> valid one while the RocksDBStateBackend will not. When using
> HeapStateBackend, if the user uses a non-deterministic shuffle key, an
> exception is thrown as follows:
>
> {code:java}
> java.lang.IllegalArgumentException: Key group 84 is not in
> KeyGroupRange{startKeyGroup=32, endKeyGroup=63}. Unless you're directly using
> low level state access APIs, this is most likely caused by non-deterministic
> shuffle key (hashCode and equals implementation).
> at
> org.apache.flink.runtime.state.KeyGroupRangeOffsets.newIllegalKeyGroupException(KeyGroupRangeOffsets.java:37)
> at
> org.apache.flink.runtime.state.heap.StateTable.getMapForKeyGroup(StateTable.java:305)
> at org.apache.flink.runtime.state.heap.StateTable.get(StateTable.java:261)
> at org.apache.flink.runtime.state.heap.StateTable.get(StateTable.java:143)
> at
> org.apache.flink.runtime.state.heap.HeapValueState.value(HeapValueState.java:72)
> at
> com.alibaba.ververica.flink.state.benchmark.wordcount.WordCount$MixedFlatMapper.flatMap(WordCount.java:169)
> at
> com.alibaba.ververica.flink.state.benchmark.wordcount.WordCount$MixedFlatMapper.flatMap(WordCount.java:160)
> at
> org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:47)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:135)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:106)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:526)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:811)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:760)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:954)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:933)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> However, the RocksDBStateBackend will run without an exception. The wrong key
> group index will cause a state correctness problem, so it is better to do a
> check in {_}InternalKeyContextImpl#{_}{_}setCurrentKeyGroupIndex{_}, and
> throw an exception immediately.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27432) Replace Time with Duration in TaskSlotTable
[ https://issues.apache.org/jira/browse/FLINK-27432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27432: - Fix Version/s: 1.17.0 (was: 1.16.0) > Replace Time with Duration in TaskSlotTable > --- > > Key: FLINK-27432 > URL: https://issues.apache.org/jira/browse/FLINK-27432 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27075) Migrate CI from Azure to Github Actions
[ https://issues.apache.org/jira/browse/FLINK-27075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27075: - Fix Version/s: 1.17.0 1.16.1 (was: 1.16.0) (was: 1.14.7) > Migrate CI from Azure to Github Actions > --- > > Key: FLINK-27075 > URL: https://issues.apache.org/jira/browse/FLINK-27075 > Project: Flink > Issue Type: Technical Debt > Components: Build System / CI >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0, 1.15.3, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26360) [Umbrella] Improvement for Hive Query Syntax Compatibility
[ https://issues.apache.org/jira/browse/FLINK-26360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-26360: - Fix Version/s: 1.17.0 (was: 1.16.0) > [Umbrella] Improvement for Hive Query Syntax Compatibility > -- > > Key: FLINK-26360 > URL: https://issues.apache.org/jira/browse/FLINK-26360 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: luoyuxia >Assignee: luoyuxia >Priority: Major > Fix For: 1.17.0 > > > Currently, we have a support for hive synatax compatibility in flink as > described in [FLINK-21529|https://issues.apache.org/jira/browse/FLINK-21529]. > But there're still some features we don't support or some other issues when > using hive synatax. > In here, we want to make a improvement to solve the issues encountered when > using Hive dialect to make it be more smoothly when you mrigate your hive job > to flink or enable you write flink job using hive synatax with less knowledge > about flink sql. > Feel free to leave your comment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29402) Add USE_DIRECT_READ configuration parameter for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29402: - Fix Version/s: 1.17.0 (was: 1.16.0) > Add USE_DIRECT_READ configuration parameter for RocksDB > --- > > Key: FLINK-29402 > URL: https://issues.apache.org/jira/browse/FLINK-29402 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Donatien >Priority: Not a Priority > Labels: Enhancement, pull-request-available, rocksdb > Fix For: 1.17.0 > > Attachments: directIO-performance-comparison.png > > Original Estimate: 1h > Remaining Estimate: 1h > > RocksDB allows the use of DirectIO for read operations to bypass the Linux > Page Cache. To understand the impact of Linux Page Cache on performance, one > can run a heavy workload on a single-tasked Task Manager with a container > memory limit identical to the TM process memory. Running this same workload > on a TM with no container memory limit will result in better performances but > with the host memory exceeding the TM requirement. > Linux Page Cache are of course useful but can give false results when > benchmarking the Managed Memory used by RocksDB. DirectIO is typically > enabled for benchmarks on working set estimation [Zwaenepoel et > al.|[https://arxiv.org/abs/1702.04323].] > I propose to add a configuration key allowing users to enable the use of > DirectIO for reads thanks to the RocksDB API. This configuration would be > disabled by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29157) Clarify the contract between CompletedCheckpointStore and SharedStateRegistry
[ https://issues.apache.org/jira/browse/FLINK-29157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29157: - Fix Version/s: 1.17.0 1.16.1 (was: 1.16.0) > Clarify the contract between CompletedCheckpointStore and SharedStateRegistry > - > > Key: FLINK-29157 > URL: https://issues.apache.org/jira/browse/FLINK-29157 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Checkpointing >Affects Versions: 1.16.0, 1.15.2 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Fix For: 1.17.0, 1.15.3, 1.16.1 > > > After FLINK-24611, CompletedCheckpointStore is required to call > SharedStateRegistry.unregisterUnusedState() on checkpoint subsumption and > shutdown. > Although it's not clear whether CompletedCheckpointStore is internal there > are in fact external implementations (which weren't updated accordingly). > > After FLINK-25872, CompletedCheckpointStore also must call > checkpointsCleaner.cleanSubsumedCheckpoints. > > Another issue with a custom implementation was using different java objects > for state for CheckpointStore and SharedStateRegistry (after FLINK-24086). > > So it makes sense to: > * clarify the contract (different in 1.15 and 1.16) > * require using the same checkpoint objects by SharedStateRegistryFactory > and CompletedCheckpointStore > * mark the interface(s) as PublicEvolving -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27077) Setup per-commit builds
[ https://issues.apache.org/jira/browse/FLINK-27077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27077: - Fix Version/s: 1.17.0 1.16.1 (was: 1.16.0) (was: 1.14.7) > Setup per-commit builds > --- > > Key: FLINK-27077 > URL: https://issues.apache.org/jira/browse/FLINK-27077 > Project: Flink > Issue Type: Sub-task > Components: Build System / CI >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0, 1.15.3, 1.16.1 > > > For initial validation, setup the per-commit builds for pushes to the master > branch. > Once validation is complete, backport to 1.14/1.15. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29298) LocalBufferPool request buffer from NetworkBufferPool hanging
[ https://issues.apache.org/jira/browse/FLINK-29298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29298: - Fix Version/s: 1.17.0 (was: 1.16.0) > LocalBufferPool request buffer from NetworkBufferPool hanging > - > > Key: FLINK-29298 > URL: https://issues.apache.org/jira/browse/FLINK-29298 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: Weijie Guo >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > Attachments: image-2022-09-14-10-52-15-259.png, > image-2022-09-14-10-58-45-987.png, image-2022-09-14-11-00-47-309.png > > > In the scenario where the buffer contention is fierce, sometimes the task > hang can be observed. Through the thread dump information, we can found that > the task thread is blocked by requestMemorySegmentBlocking forever. After > investigating the dumped heap information, I found that the NetworkBufferPool > actually has many buffers, but the LocalBufferPool is still unavailable and > no buffer has been obtained. > By looking at the code, I am sure that this is a bug in thread race: when the > task thread polled out the last buffer in LocalBufferPool and triggered the > onGlobalPoolAvailable callback itself, it will skip this notification (as > currently the LocalBufferPool is available), which will cause the BufferPool > to eventually become unavailable and will never register a callback to the > NetworkBufferPool. > The conditions for triggering the problem are relatively strict, but I have > found a stable way to reproduce it, I will try to fix and verify this problem. > !image-2022-09-14-10-52-15-259.png|width=1021,height=219! > !image-2022-09-14-10-58-45-987.png|width=997,height=315! > !image-2022-09-14-11-00-47-309.png|width=453,height=121! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27753) Scala suffix checks should check for maven errors
[ https://issues.apache.org/jira/browse/FLINK-27753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27753: - Fix Version/s: 1.17.0 (was: 1.16.0) > Scala suffix checks should check for maven errors > - > > Key: FLINK-27753 > URL: https://issues.apache.org/jira/browse/FLINK-27753 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Build System / CI >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > > As shown in FLINK-27751 errors in maven when retrieving the dependency tree > are currently ignored, causing garbage output being passed to the suffix > checker. > This results in highly misleading error messages about incorrect suffixes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27455) [JUnit5 Migration] SnapshotMigrationTestBase
[ https://issues.apache.org/jira/browse/FLINK-27455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27455: - Fix Version/s: 1.17.0 (was: 1.16.0) > [JUnit5 Migration] SnapshotMigrationTestBase > > > Key: FLINK-27455 > URL: https://issues.apache.org/jira/browse/FLINK-27455 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27299) flink parsing parameter bug
[ https://issues.apache.org/jira/browse/FLINK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-27299: - Fix Version/s: 1.17.0 (was: 1.16.0) > flink parsing parameter bug > --- > > Key: FLINK-27299 > URL: https://issues.apache.org/jira/browse/FLINK-27299 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.11.6, 1.13.5, 1.14.2, 1.13.6, 1.14.3, 1.14.4 >Reporter: Huajie Wang >Assignee: Huajie Wang >Priority: Minor > Labels: easyfix, pull-request-available, stale-assigned > Fix For: 1.17.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > When I am running a flink job, I specify a running parameter with a "#" sign > in it. The parsing fails. > e.g: flink run com.myJob --sink.password db@123#123 > only parse the content in front of "#", after reading the source code It is > found that the parameters are intercepted according to "#" in the > loadYAMLResource method of GlobalConfiguration. This part needs to be improved -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28960) Pulsar throws java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlElement
[
https://issues.apache.org/jira/browse/FLINK-28960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-28960:
-
Fix Version/s: 1.17.0
1.16.1
(was: 1.16.0)
> Pulsar throws java.lang.NoClassDefFoundError:
> javax/xml/bind/annotation/XmlElement
> --
>
> Key: FLINK-28960
> URL: https://issues.apache.org/jira/browse/FLINK-28960
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.15.1, 1.14.6
>Reporter: Yufan Sheng
>Priority: Minor
> Fix For: 1.17.0, 1.15.3, 1.14.7, 1.16.1
>
>
> {code:java}
> Unknown HK2 failure detected:
> MultiException stack 1 of 2
> java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlElement
> at
> org.apache.pulsar.shade.com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.(JaxbAnnotationIntrospector.java:137)
> at
> org.apache.pulsar.shade.com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.(JaxbAnnotationIntrospector.java:124)
> at
> org.apache.pulsar.shade.com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.(JaxbAnnotationIntrospector.java:116)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at java.base/java.lang.Class.newInstance(Class.java:584)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-25612) Update the outdated illustration of ExecutionState in the documentation
[
https://issues.apache.org/jira/browse/FLINK-25612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-25612:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Update the outdated illustration of ExecutionState in the documentation
> ---
>
> Key: FLINK-25612
> URL: https://issues.apache.org/jira/browse/FLINK-25612
> Project: Flink
> Issue Type: Technical Debt
> Components: Documentation
>Affects Versions: 1.13.5, 1.14.2, 1.15.0
>Reporter: Zhilong Hong
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.0
>
> Attachments: current-illustration-2.jpg, new-illustration-2.jpg
>
>
> Currently, the illustration of {{ExecutionState}} located in the page "Jobs
> and Scheduling"
> ([https://nightlies.apache.org/flink/flink-docs-master/docs/internals/job_scheduling/])
> is outdated. It doesn't involve the INITIALIZING state, which is introduced
> in FLINK-17102.
>
> Current illustration:
> !current-illustration-2.jpg|width=400!
> New illustration:
> !new-illustration-2.jpg|width=400!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24943) SequenceNumber class is not POJO type
[
https://issues.apache.org/jira/browse/FLINK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-24943:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> SequenceNumber class is not POJO type
> -
>
> Key: FLINK-24943
> URL: https://issues.apache.org/jira/browse/FLINK-24943
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.12.5, 1.13.3, 1.14.4, 1.15.0
>Reporter: Alexander Egorov
>Assignee: Alexander Egorov
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> SequenceNumber class is currently part of the "Kinesis-Stream-Shard-State",
> but it does not follow requirements of POJO compatible type. Because of that
> we are getting warning like this:
> {{TypeExtractor - class
> software.amazon.kinesis.connectors.flink.model.SequenceNumber does not
> contain a setter for field sequenceNumber}}
> While the warning itself or inability to use optimal sterilizer for such a
> small state is not the problem, this warning prevents us to disable Generic
> Types via {{disableGenericTypes()}}
> So, the problem is similar to
> https://issues.apache.org/jira/browse/FLINK-15904
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28553) The serializer in StateMap has not been updated when metaInfo of StateTable updated
[ https://issues.apache.org/jira/browse/FLINK-28553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-28553: - Fix Version/s: 1.17.0 (was: 1.16.0) > The serializer in StateMap has not been updated when metaInfo of StateTable > updated > --- > > Key: FLINK-28553 > URL: https://issues.apache.org/jira/browse/FLINK-28553 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.14.5, 1.15.1 >Reporter: Hangxiang Yu >Assignee: Hangxiang Yu >Priority: Minor > Fix For: 1.17.0 > > > When the meta info in StateTable updated, the serializer in StateMap has not > been updated. > (See StateTable#setMetaInfo) > The value may be serialized/deserialized/copied incorrectly after triggering > state migration of HashMapStateBackend. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-25842: - Fix Version/s: 1.17.0 (was: 1.16.0) > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.17.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28766) UnalignedCheckpointStressITCase.runStressTest failed with NoSuchFileException
[
https://issues.apache.org/jira/browse/FLINK-28766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-28766:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> UnalignedCheckpointStressITCase.runStressTest failed with NoSuchFileException
> -
>
> Key: FLINK-28766
> URL: https://issues.apache.org/jira/browse/FLINK-28766
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Anton Kalashnikov
>Priority: Critical
> Labels: test-stability
> Fix For: 1.17.0
>
>
> {code:java}
> 2022-08-01T01:36:16.0563880Z Aug 01 01:36:16 [ERROR]
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runStressTest
> Time elapsed: 12.579 s <<< ERROR!
> 2022-08-01T01:36:16.0565407Z Aug 01 01:36:16 java.io.UncheckedIOException:
> java.nio.file.NoSuchFileException:
> /tmp/junit1058240190382532303/f0f99754a53d2c4633fed75011da58dd/chk-7/61092e4a-5b9a-4f56-83f7-d9960c53ed3e
> 2022-08-01T01:36:16.0566296Z Aug 01 01:36:16 at
> java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
> 2022-08-01T01:36:16.0566972Z Aug 01 01:36:16 at
> java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
> 2022-08-01T01:36:16.0567600Z Aug 01 01:36:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2022-08-01T01:36:16.0568290Z Aug 01 01:36:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2022-08-01T01:36:16.0569172Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2022-08-01T01:36:16.0569877Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> 2022-08-01T01:36:16.0570554Z Aug 01 01:36:16 at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> 2022-08-01T01:36:16.0571371Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 2022-08-01T01:36:16.0572417Z Aug 01 01:36:16 at
> java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:546)
> 2022-08-01T01:36:16.0573618Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.discoverRetainedCheckpoint(UnalignedCheckpointStressITCase.java:289)
> 2022-08-01T01:36:16.0575187Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runAndTakeExternalCheckpoint(UnalignedCheckpointStressITCase.java:262)
> 2022-08-01T01:36:16.0576540Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runStressTest(UnalignedCheckpointStressITCase.java:158)
> 2022-08-01T01:36:16.0577684Z Aug 01 01:36:16 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-01T01:36:16.0578546Z Aug 01 01:36:16 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-01T01:36:16.0579374Z Aug 01 01:36:16 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-01T01:36:16.0580298Z Aug 01 01:36:16 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-01T01:36:16.0581243Z Aug 01 01:36:16 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-08-01T01:36:16.0582029Z Aug 01 01:36:16 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-08-01T01:36:16.0582766Z Aug 01 01:36:16 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-08-01T01:36:16.0583488Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-08-01T01:36:16.0584203Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-08-01T01:36:16.0585087Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-08-01T01:36:16.0585778Z Aug 01 01:36:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-08-01T01:36:16.0586482Z Aug 01 01:36:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-08-01T01:36:16.0587155Z Aug 01 01:36:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-08-01T01:36:16.0587809Z Aug 01 01:36:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-08-01T01:36:16.0588434Z Aug 01 01:36:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-08-01T01:36:16.0589203Z Aug 01 01:36:16 at
>
[jira] [Updated] (FLINK-29179) Translate "Temporal Table Function" page of "Streaming Concepts" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-29179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29179: - Fix Version/s: 1.17.0 (was: 1.16.0) > Translate "Temporal Table Function" page of "Streaming Concepts" into Chinese > - > > Key: FLINK-29179 > URL: https://issues.apache.org/jira/browse/FLINK-29179 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Affects Versions: 1.15.2 >Reporter: ChunJi >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > The page url is > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/concepts/temporal_table_function/] > The markdown file is located in > docs/content.zh/docs/dev/table/concepts/temporal_table_function.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25862) Refactor SharedStateRegistry to not limit StreamStateHandle to register/unregister
[ https://issues.apache.org/jira/browse/FLINK-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-25862: - Fix Version/s: 1.17.0 (was: 1.16.0) > Refactor SharedStateRegistry to not limit StreamStateHandle to > register/unregister > -- > > Key: FLINK-25862 > URL: https://issues.apache.org/jira/browse/FLINK-25862 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: Yun Tang >Priority: Minor > Fix For: 1.17.0 > > > Current implementation of SharedStateRegistry would use `StreamStateHandle` > to register and unregister. This would limit the usage for other componments, > such as change-log state backend handle usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26369) Translate the part zh-page mixed with not be translated.
[ https://issues.apache.org/jira/browse/FLINK-26369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-26369: - Fix Version/s: 1.17.0 (was: 1.16.0) > Translate the part zh-page mixed with not be translated. > - > > Key: FLINK-26369 > URL: https://issues.apache.org/jira/browse/FLINK-26369 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Aiden Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > These file should be translated. > Files: > docs/content.zh/docs/deployment/ha/overview.md > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-24932:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
> Fix For: 1.17.0
>
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
>
[jira] [Updated] (FLINK-23035) Add explicit method to StateChangelogWriter to write metadata
[ https://issues.apache.org/jira/browse/FLINK-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-23035: - Fix Version/s: 1.17.0 (was: 1.16.0) > Add explicit method to StateChangelogWriter to write metadata > - > > Key: FLINK-23035 > URL: https://issues.apache.org/jira/browse/FLINK-23035 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Xinbin Huang >Priority: Minor > Fix For: 1.17.0 > > > Currently, metadata is written to the state changelog using the same > StateChangelogWriter.append() method as data. > However, it doesn't belong to a specific group, and should be read first on > recovery. Because of that, -1 is used. > An explicit append() without keygroup would be less fragile (probably still > using -1 under the hood). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24688) yarn.application-attempt-failures-validity-interval link url is invalid
[ https://issues.apache.org/jira/browse/FLINK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-24688: - Fix Version/s: 1.17.0 (was: 1.16.0) > yarn.application-attempt-failures-validity-interval link url is invalid > --- > > Key: FLINK-24688 > URL: https://issues.apache.org/jira/browse/FLINK-24688 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0 >Reporter: baizhendong >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > yarn.application-attempt-failures-validity-interval property was added in > YarnConfigOptions since FLINK-12472[yarn], but now the link > [https://hortonworks.com/blog/apache-hadoop-yarn-hdp-2-2-fault-tolerance-features-long-running-services/] > is not available. So we should update this link. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24497) Kafka metrics fetching throws IllegalStateException
[
https://issues.apache.org/jira/browse/FLINK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-24497:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Kafka metrics fetching throws IllegalStateException
>
>
> Key: FLINK-24497
> URL: https://issues.apache.org/jira/browse/FLINK-24497
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Juha
>Priority: Minor
> Fix For: 1.17.0
>
>
> I have a simple job that just consumes from a single Kafka topic, performs
> some filtering and produces to another topic.
> The TaskManager log has these periodically. This is a new problem in 1.14.0,
> the same setup didn't have the issue when using 1.13.0 or 1.13.2.
> {code}
> 2021-10-05T15:22:31.928316 [2021-10-05 15:22:31,927] WARN Error when getting
> Kafka consumer metric "records-lag" for partition "cpu.kafka-1". Metric
> "pendingBytes" may not be reported correctly.
> (org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics:306)
> 2021-10-05T15:22:31.928316 java.lang.IllegalStateException: Cannot find
> Kafka metric matching current filter.
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.MetricUtil.lambda$getKafkaMetric$1(MetricUtil.java:63)
> ~[flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> java.util.Optional.orElseThrow(Optional.java:408) ~[?:?]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.MetricUtil.getKafkaMetric(MetricUtil.java:61)
> ~[flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics.getRecordsLagMetric(KafkaSourceReaderMetrics.java:304)
> ~[flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics.lambda$maybeAddRecordsLagMetric$4(KafkaSourceReaderMetrics.java:229)
> ~[flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
> [?:?]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics.maybeAddRecordsLagMetric(KafkaSourceReaderMetrics.java:228)
> [flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader.fetch(KafkaPartitionSplitReader.java:187)
> [flink-connector-kafka_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
> [flink-table_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
> [flink-table_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
> [flink-table_2.12-1.14.0.jar:1.14.0]
> 2021-10-05T15:22:31.928316at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> 2021-10-05T15:22:31.928316at
> java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> 2021-10-05T15:22:31.928316at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> 2021-10-05T15:22:31.928316at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> 2021-10-05T15:22:31.928316at java.lang.Thread.run(Thread.java:829) [?:?]
> {code}
> Regards,
> Juha
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-22842) Streaming File Sink end-to-end test fail due to not finish after 900s
[ https://issues.apache.org/jira/browse/FLINK-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-22842. Resolution: Cannot Reproduce > Streaming File Sink end-to-end test fail due to not finish after 900s > - > > Key: FLINK-22842 > URL: https://issues.apache.org/jira/browse/FLINK-22842 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.14.0 >Reporter: Xintong Song >Priority: Minor > Labels: auto-deprioritized-major, test-stability > Fix For: 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18525=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=13152 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-22334) Fail to translate the hive-sql in STREAMING mode
[
https://issues.apache.org/jira/browse/FLINK-22334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-22334:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Fail to translate the hive-sql in STREAMING mode
>
>
> Key: FLINK-22334
> URL: https://issues.apache.org/jira/browse/FLINK-22334
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Shengkai Fang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.17.0
>
>
> Please run in the streaming mode.
> The failed statement
> {code:java}
> // Some comments here
> insert into dest(y,x) select x,y from foo cluster by x
> {code}
> Exception stack:
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties: convention=LOGICAL,
> FlinkRelDistributionTraitDef=any, MiniBatchIntervalTraitDef=None: 0,
> ModifyKindSetTraitDef=[NONE], UpdateKindTraitDef=[NONE].
> Missing conversion is LogicalDistribution[convention: NONE -> LOGICAL]
> There is 1 empty subset: rel#5176:RelSubset#43.LOGICAL.any.None:
> 0.[NONE].[NONE], the relevant part of the original plan is as follows
> 5174:LogicalDistribution(collation=[[0 ASC-nulls-first]], dist=[[]])
> 5172:LogicalProject(subset=[rel#5173:RelSubset#42.NONE.any.None:
> 0.[NONE].[NONE]], x=[$0])
> 5106:LogicalTableScan(subset=[rel#5171:RelSubset#41.NONE.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, foo]])
> Root: rel#5176:RelSubset#43.LOGICAL.any.None: 0.[NONE].[NONE]
> Original rel:
> FlinkLogicalLegacySink(subset=[rel#4254:RelSubset#8.LOGICAL.any.None:
> 0.[NONE].[NONE]], name=[collect], fields=[_o__c0]): rowcount = 1.0E8,
> cumulative cost = {1.0E8 rows, 1.0E8 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 4276
> FlinkLogicalCalc(subset=[rel#4275:RelSubset#7.LOGICAL.any.None:
> 0.[NONE].[NONE]], select=[CASE(IS NULL($f1), 0:BIGINT, $f1) AS _o__c0]):
> rowcount = 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 4288
> FlinkLogicalJoin(subset=[rel#4272:RelSubset#6.LOGICAL.any.None:
> 0.[NONE].[NONE]], condition=[=($0, $1)], joinType=[left]): rowcount = 1.0E8,
> cumulative cost = {1.0E8 rows, 1.0856463237676364E8 cpu, 4.0856463237676364E8
> io, 0.0 network, 0.0 memory}, id = 4271
>
> FlinkLogicalTableSourceScan(subset=[rel#4270:RelSubset#1.LOGICAL.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, bar, project=[i]]], fields=[i]):
> rowcount = 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 4.0E8 io, 0.0
> network, 0.0 memory}, id = 4279
> FlinkLogicalAggregate(subset=[rel#4268:RelSubset#5.LOGICAL.any.None:
> 0.[NONE].[NONE]], group=[{1}], agg#0=[COUNT($0)]): rowcount =
> 8564632.376763644, cumulative cost = {9.0E7 rows, 1.89E8 cpu, 7.2E8 io, 0.0
> network, 0.0 memory}, id = 4286
> FlinkLogicalCalc(subset=[rel#4283:RelSubset#3.LOGICAL.any.None:
> 0.[NONE].[NONE]], select=[x, y], where=[IS NOT NULL(y)]): rowcount = 9.0E7,
> cumulative cost = {9.0E7 rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
> = 4282
>
> FlinkLogicalTableSourceScan(subset=[rel#4262:RelSubset#2.LOGICAL.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, foo]], fields=[x, y]): rowcount =
> 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 8.0E8 io, 0.0 network, 0.0
> memory}, id = 4261
> Sets:
> Set#41, type: RecordType(INTEGER x, INTEGER y)
> rel#5171:RelSubset#41.NONE.any.None: 0.[NONE].[NONE], best=null
> rel#5106:LogicalTableScan.NONE.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo]), rowcount=1.0E8, cumulative
> cost={inf}
> rel#5179:RelSubset#41.LOGICAL.any.None: 0.[NONE].[NONE], best=rel#5178
> rel#5178:FlinkLogicalTableSourceScan.LOGICAL.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo],fields=x, y), rowcount=1.0E8,
> cumulative cost={1.0E8 rows, 1.0E8 cpu, 8.0E8 io, 0.0 network, 0.0 memory}
> Set#42, type: RecordType(INTEGER x)
> rel#5173:RelSubset#42.NONE.any.None: 0.[NONE].[NONE], best=null
> rel#5172:LogicalProject.NONE.any.None:
> 0.[NONE].[NONE](input=RelSubset#5171,inputs=0), rowcount=1.0E8, cumulative
> cost={inf}
> rel#5180:LogicalTableScan.NONE.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo, project=[x]]), rowcount=1.0E8,
> cumulative cost={inf}
> rel#5182:LogicalCalc.NONE.any.None:
> 0.[NONE].[NONE](input=RelSubset#5171,expr#0..1={inputs},0=$t0),
> rowcount=1.0E8, cumulative cost={inf}
> rel#5184:RelSubset#42.LOGICAL.any.None: 0.[NONE].[NONE], best=rel#5183
> rel#5183:FlinkLogicalTableSourceScan.LOGICAL.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo,
[jira] [Updated] (FLINK-21125) Sum or Sum0 overflow quietly
[
https://issues.apache.org/jira/browse/FLINK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-21125:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Sum or Sum0 overflow quietly
>
>
> Key: FLINK-21125
> URL: https://issues.apache.org/jira/browse/FLINK-21125
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Sebastian Liu
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.17.0
>
>
> Overflow is not calculated correctly in the build-in sum function of Blink
> planner.
> For a aggregate calculation such as:
> {code:java}
> CREATE TABLE TestTable (
> amount INT
> );
> insert into TestTable (2147483647);
> insert into TestTable (1);
> SELECT sum(amount) FROM TestTable;
> The result will be: -2147483648, which is an overflowed value and no
> exception was thrown. {code}
> The overflow occurs quietly and is difficult to detect.
> Compare the processing semantics of other systems：
> * *mysql*: provide two SQL mode for handling overflow. If strict SQL mode is
> enabled, MySQL rejects the out-of-range value with an error, and the insert
> fails, in accordance with the SQL standard. If no restrictive modes are
> enabled, MySQL clips the value to the appropriate endpoint of the column data
> type range and stores the resulting value instead. FYI:
> [https://dev.mysql.com/doc/refman/8.0/en/out-of-range-and-overflow.html]
> * *presto*: all numeric types are automatically cast to Long type, and If
> the long is out of range, an exception is thrown to prompt user.
> IMO, exception hint is necessary, instead of quietly overflow.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-21765) Remove implementation-specific MetricGroup parents
[ https://issues.apache.org/jira/browse/FLINK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-21765: - Fix Version/s: 1.17.0 (was: 1.16.0) > Remove implementation-specific MetricGroup parents > -- > > Key: FLINK-21765 > URL: https://issues.apache.org/jira/browse/FLINK-21765 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > stale-assigned, starter > Fix For: 1.17.0 > > > MetricGroups currently form a bi-directly graph, usually with explicit > requirements that type the parent must have. For example, an OperatorMG has a > hard requirement that the parent is a TaskMG. > As a result they are quite inflexible, which particular shows in tests, as > you can't just create one metric group, but have to build an entire tree. > The end goal of this ticket is to remove AbstractMetricGroup#parent, and > along the way we'll decouple the various MG implementations from each other. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20818) End to end test produce excessive amount of logs
[ https://issues.apache.org/jira/browse/FLINK-20818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-20818: - Fix Version/s: 1.17.0 (was: 1.16.0) > End to end test produce excessive amount of logs > > > Key: FLINK-20818 > URL: https://issues.apache.org/jira/browse/FLINK-20818 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Affects Versions: 1.13.0 >Reporter: Till Rohrmann >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.17.0 > > > The end to end test produce an excessive amount of logs. For example in this > run [1] the log file is roughly 57 MB and it is no longer possible to > properly scroll in this file when using the web interface. I think there > should not be a reason for producing almost 60 MB of log output. > [1] > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11467=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20155) java.lang.OutOfMemoryError: Direct buffer memory
[
https://issues.apache.org/jira/browse/FLINK-20155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-20155:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> java.lang.OutOfMemoryError: Direct buffer memory
>
>
> Key: FLINK-20155
> URL: https://issues.apache.org/jira/browse/FLINK-20155
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.11.1
>Reporter: roee hershko
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.17.0
>
> Attachments: image-2020-11-13-17-52-54-217.png
>
>
> update:
> this issue occur every time after a job fails the only way to fix it is to
> manually re-create the task managers pods (i am using flink operator)
>
> after submitting a job, it runs for few hours and then the job manager is
> crushing, when trying to re-create the job i am getting the following error:
>
> {code:java}
> 2020-11-13 17:44:58org.apache.pulsar.client.admin.PulsarAdminException:
> org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memoryat
> org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:228)
> at
> org.apache.pulsar.client.admin.internal.TopicsImpl$7.failed(TopicsImpl.java:324)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:231)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:85)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:183)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:183)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onThrowable(AsyncHttpConnector.java:279)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:277)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.WriteListener.abortOnThrowable(WriteListener.java:50)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.WriteListener.operationComplete(WriteListener.java:61)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.WriteCompleteListener.operationComplete(WriteCompleteListener.java:28)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.WriteCompleteListener.operationComplete(WriteCompleteListener.java:20)
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:421)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener.writeRequest(NettyConnectListener.java:80)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:156)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:92)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:26)
> at
>
[jira] [Updated] (FLINK-19562) the /flink/docs/docker/run.sh cause an error with numeric username
[
https://issues.apache.org/jira/browse/FLINK-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-19562:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> the /flink/docs/docker/run.sh cause an error with numeric username
> --
>
> Key: FLINK-19562
> URL: https://issues.apache.org/jira/browse/FLINK-19562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.11.2
>Reporter: shizhengchao
>Priority: Minor
> Fix For: 1.17.0
>
>
> if the user of the OS is a number, run the script and report an error:
> {code:java}
> groupadd: '002117' is not a valid group name
> {code}
> if the user of the OS is english, will be ok, like this:
> {code:java}
> if [ "$(uname -s)" == "Linux" ]; then
> USER_NAME=${SUDO_USER:=$USER}
> USER_ID=$(id -u "${USER_NAME}")
> GROUP_ID=$(id -g "${USER_NAME}")
> else # boot2docker uid and gid
> USER_NAME='mwopr'
> USER_ID=1000
> GROUP_ID=50
> fi
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-20305) Iterative job with maxWaitTimeMillis does not terminate
[
https://issues.apache.org/jira/browse/FLINK-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-20305:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Iterative job with maxWaitTimeMillis does not terminate
> ---
>
> Key: FLINK-20305
> URL: https://issues.apache.org/jira/browse/FLINK-20305
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
> Fix For: 1.17.0
>
>
> While testing the {{DataStream}} API, I noticed that an iterative job with
> {{maxWaitTimeMillis}} set, sometimes did not terminate depending on the used
> parallelism. Since I used a bounded source with few events, it looked a bit
> as if the iteration tasks couldn't terminate if they didn't receive an event.
> The corresponding testing job including test data can be found
> [here|https://github.com/tillrohrmann/flink-streaming-batch-execution/tree/iteration-problem].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-23869) Correct javadoc of ExecutionConfig#registerKryoType
[ https://issues.apache.org/jira/browse/FLINK-23869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-23869: - Fix Version/s: 1.17.0 (was: 1.16.0) > Correct javadoc of ExecutionConfig#registerKryoType > --- > > Key: FLINK-23869 > URL: https://issues.apache.org/jira/browse/FLINK-23869 > Project: Flink > Issue Type: Bug > Components: API / Core, Documentation >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Fix For: 1.17.0 > > > Current javadoc of ExecutionConfig#registerKryoType is mistakely copied from > ExecutionConfig#registerPojoType, we could fix this error by copying from > documentation, -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-22826) flink sql1.13.1 causes data loss based on change log stream data join
[
https://issues.apache.org/jira/browse/FLINK-22826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-22826:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> flink sql1.13.1 causes data loss based on change log stream data join
> -
>
> Key: FLINK-22826
> URL: https://issues.apache.org/jira/browse/FLINK-22826
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0, 1.13.1
>Reporter: 徐州州
>Priority: Minor
> Labels: auto-deprioritized-major, stale-blocker
> Fix For: 1.17.0
>
>
> {code:java}
> insert into dwd_order_detail
> select
>ord.Id,
>ord.Code,
>Status
> concat(cast(ord.Id as String),if(oed.Id is null,'oed_null',cast(oed.Id
> as STRING)),DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd')) as uuids,
> TO_DATE(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd')) as As_Of_Date
> from
> orders ord
> left join order_extend oed on ord.Id=oed.OrderId and oed.IsDeleted=0 and
> oed.CreateTime>CAST(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd') AS TIMESTAMP)
> where ( ord.OrderTime>CAST(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd') AS
> TIMESTAMP)
> or ord.ReviewTime>CAST(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd') AS TIMESTAMP)
> or ord.RejectTime>CAST(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd') AS TIMESTAMP)
> ) and ord.IsDeleted=0;
> {code}
> My upsert-kafka table for PRIMARY KEY for uuids.
> This is the logic of my kafka based canal-json stream data join and write to
> Upsert-kafka tables I confirm that version 1.12 also has this problem I just
> upgraded from 1.12 to 1.13.
> I look up a user s order data and order number XJ0120210531004794 in
> canal-json original table as U which is normal.
> {code:java}
> | +U | XJ0120210531004794 | 50 |
> | +U | XJ0120210531004672 | 50 |
> {code}
> But written to upsert-kakfa via join, the data consumed from upsert kafka is,
> {code:java}
> | +I | XJ0120210531004794 | 50 |
> | -U | XJ0120210531004794 | 50 |
> {code}
> The order is two records this sheet in orders and order_extend tables has not
> changed since created -U status caused my data loss not computed and the
> final result was wrong.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-22567) OrcFsStreamingSinkITCase crashes JVM
[
https://issues.apache.org/jira/browse/FLINK-22567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-22567.
Resolution: Cannot Reproduce
> OrcFsStreamingSinkITCase crashes JVM
>
>
> Key: FLINK-22567
> URL: https://issues.apache.org/jira/browse/FLINK-22567
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ORC, Formats (JSON, Avro, Parquet, ORC,
> SequenceFile)
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
> Fix For: 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17552=logs=e1276d0f-df12-55ec-86b5-c0ad597d83c9=906e9244-f3be-5604-1979-e767c8a6f6d9=12508
> {code}
> May 04 23:51:48 [ERROR] Process Exit Code: 239
> May 04 23:51:48 [ERROR] Crashed tests:
> May 04 23:51:48 [ERROR] org.apache.flink.orc.OrcFsStreamingSinkITCase
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> May 04 23:51:48 [ERROR] at
> org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> May 04 23:51:48 [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> May 04 23:51:48 [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> May 04 23:51:48 [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> May 04 23:51:48 [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> May 04 23:51:48 [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> May 04 23:51:48 [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> May 04 23:51:48 [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> May 04 23:51:48 [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> May 04 23:51:48 [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> May 04 23:51:48 [ERROR] Command was /bin/sh -c cd
> /__w/1/s/flink-formats/flink-orc/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/1/s/flink-formats/flink-orc/target/surefire/surefirebooter2066197408916384359.jar
> /__w/1/s/flink-formats/flink-orc/target/surefire
> 2021-05-04T23-33-56_338-jvmRun2 surefire8400240753057854698tmp
> surefire_512690828581116685255tmp
> May 04 23:51:48 [ERROR] Error
[jira] [Updated] (FLINK-21307) Revisit activation model of FlinkSecurityManager
[ https://issues.apache.org/jira/browse/FLINK-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-21307: - Fix Version/s: 1.17.0 (was: 1.16.0) > Revisit activation model of FlinkSecurityManager > > > Key: FLINK-21307 > URL: https://issues.apache.org/jira/browse/FLINK-21307 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.13.0 >Reporter: Robert Metzger >Priority: Minor > Labels: auto-deprioritized-critical, auto-deprioritized-major > Fix For: 1.17.0 > > > In FLINK-15156, we introduced a feature that allows users to log or > completely disable calls to System.exit(). This feature is enabled for > certain threads / code sections intended to execute user-code. > The activation of the security manager (for monitoring user calls to > System.exit() is currently not well-defined, and only implemented on a > best-effort basis. > This ticket is to revisit the activation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-21898) JobRetrievalITCase crash
[ https://issues.apache.org/jira/browse/FLINK-21898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo closed FLINK-21898. Resolution: Cannot Reproduce > JobRetrievalITCase crash > > > Key: FLINK-21898 > URL: https://issues.apache.org/jira/browse/FLINK-21898 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.13.0 >Reporter: Guowei Ma >Priority: Minor > Labels: auto-deprioritized-major, test-stability > Fix For: 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15083=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228=4383 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-21613) Parse Compute Column with `IN` expression throws NPE
[
https://issues.apache.org/jira/browse/FLINK-21613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-21613:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Parse Compute Column with `IN` expression throws NPE
>
>
> Key: FLINK-21613
> URL: https://issues.apache.org/jira/browse/FLINK-21613
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Shuo Cheng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
> Fix For: 1.17.0
>
>
> Considering the following given sql:
> {code:sql}
> CREATE TABLE MyInputFormatTable (
> `a` INT,
> `b` BIGINT,
> `c` STRING,
> `d` as `c` IN ('Hi', 'Hello')
> ) WITH (
> 'connector' = 'values',
> 'data-id' = '$dataId',
> 'runtime-source' = 'InputFormat'
> )
> {code}
> NPE will be thrown during parsing the sql:
> `select * from MyInputFormatTable`
> It seems it's the commit "[hotfix][table-planner-blink] Simplify SQL
> expression to RexNode conversion" which introduces this problem. This hotfix
> uses a method `SqlToRelConverter#convertExpression` and this method does not
> has any tests and is not used in Calcite anywhere, which is unsafe. Maybe
> reverting the hotfix is a good choice.
> CC [~twalthr]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29483) flink python udf arrow in thread model bug
[ https://issues.apache.org/jira/browse/FLINK-29483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29483: - Fix Version/s: 1.16.1 (was: 1.16.0) > flink python udf arrow in thread model bug > -- > > Key: FLINK-29483 > URL: https://issues.apache.org/jira/browse/FLINK-29483 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.16.0, 1.15.2 >Reporter: jackylau >Priority: Major > Fix For: 1.17.0, 1.15.3, 1.16.1 > > Attachments: image-2022-09-30-17-03-05-005.png > > > !image-2022-09-30-17-03-05-005.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-19254) Invalid UTF-8 start byte exception
[
https://issues.apache.org/jira/browse/FLINK-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-19254:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Invalid UTF-8 start byte exception
> ---
>
> Key: FLINK-19254
> URL: https://issues.apache.org/jira/browse/FLINK-19254
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.11.0
>Reporter: Jun Zhang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.17.0
>
>
> when read no utf8 data ,JsonRowDeserializationSchema throw a exception.
> {code:java}
> Caused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
> Invalid UTF-8 start byte xxx
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-19482) OrcRowInputFormat does not define serialVersionUID
[
https://issues.apache.org/jira/browse/FLINK-19482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-19482:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> OrcRowInputFormat does not define serialVersionUID
> --
>
> Key: FLINK-19482
> URL: https://issues.apache.org/jira/browse/FLINK-19482
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.17.0
>
>
> The org.apache.flink.orc function does not define a {{serialVersionUID}}.
> We should define a {{serialVersionUID when object is serialized to avoid
> [java.io|http://java.io/].InvalidClassExceptio exception.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-18777) Supports schema registry catalog
[ https://issues.apache.org/jira/browse/FLINK-18777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-18777: - Fix Version/s: 1.17.0 (was: 1.16.0) > Supports schema registry catalog > > > Key: FLINK-18777 > URL: https://issues.apache.org/jira/browse/FLINK-18777 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.0 >Reporter: Danny Chen >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.17.0 > > > Design doc: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-125%3A+Confluent+Schema+Registry+Catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-17482) KafkaITCase.testMultipleSourcesOnePartition unstable
[
https://issues.apache.org/jira/browse/FLINK-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-17482.
Resolution: Cannot Reproduce
> KafkaITCase.testMultipleSourcesOnePartition unstable
>
>
> Key: FLINK-17482
> URL: https://issues.apache.org/jira/browse/FLINK-17482
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Fabian Paul
>Priority: Minor
> Labels: auto-deprioritized-major, stale-assigned, test-stability
> Fix For: 1.16.0
>
>
> CI run:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=454=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 07:29:40,472 [main] INFO
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase[] -
> -
> [ERROR] Tests run: 22, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 152.018 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> [ERROR]
> testMultipleSourcesOnePartition(org.apache.flink.streaming.connectors.kafka.KafkaITCase)
> Time elapsed: 4.257 s <<< FAILURE!
> java.lang.AssertionError: Test failed: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:45)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:963)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testMultipleSourcesOnePartition(KafkaITCase.java:107)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
