[GitHub] [flink] flinkbot edited a comment on pull request #16243: [FLINK-23065][table-api-java][planner-blink] Introduce TableEnvironment#createTemporaryTable
flinkbot edited a comment on pull request #16243: URL: https://github.com/apache/flink/pull/16243#issuecomment-865968333 ## CI report: * e81dd1f0e200531eb6e427da1929ca6479da49c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19318) * 4880ee885e1aac73edac8053113b811d647bf294 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16220: [FLINK-23033][python] Add the missing ObjectArrayTypeInfo
flinkbot edited a comment on pull request #16220: URL: https://github.com/apache/flink/pull/16220#issuecomment-865041649 ## CI report: * ee3f8975908e9f243bb5510d0cb09a8ddf341f7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19315) * cf469ce915e4f73fbd67d24c96d6190b113ec525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19344) * 702e873d300360fe3c6a74a6c5aa9af01152cf9c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19350) * 463920ee3bdd4b7b4322fe3f5969e68bf5516266 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16146: [FLINK-22927][python] Fix the bug of JobStatus
flinkbot edited a comment on pull request #16146: URL: https://github.com/apache/flink/pull/16146#issuecomment-859379229 ## CI report: * ce5eeaa31e866ed8950567b10ec973f85aef8d7a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19343) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-jira-bot] knaufk merged pull request #18: [hotfix] add missing "days" after number of days
knaufk merged pull request #18: URL: https://github.com/apache/flink-jira-bot/pull/18 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14839: [FLINK-21353][state] Add DFS-based StateChangelog
flinkbot edited a comment on pull request #14839: URL: https://github.com/apache/flink/pull/14839#issuecomment-772060196 ## CI report: * d6adb94d3584e2d1a66e54178a006b36aea1970c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19341) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #16243: [FLINK-23065][table-api-java][planner-blink] Introduce TableEnvironment#createTemporaryTable
Airblader commented on a change in pull request #16243: URL: https://github.com/apache/flink/pull/16243#discussion_r656774209 ## File path: flink-python/pyflink/table/tests/test_environment_completeness.py ## @@ -41,6 +41,7 @@ def excluded_methods(cls): 'getCompletionHints', 'fromValues', 'create', +'createTemporaryTable', Review comment: Is it worth exposing these through PyFlink as well? If I understand correctly that requires replicating the (Table/Format)Descriptor classes in Python, right? If we want to do this I would prefer opening another sub-task for the FLIP where we expose all methods for PyFlink since it'll mostly be pass-through work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Zakelly commented on a change in pull request #15420: [FLINK-21356] Implement incremental checkpointing and recovery using state changelog
Zakelly commented on a change in pull request #15420:
URL: https://github.com/apache/flink/pull/15420#discussion_r656768464
##
File path:
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##
@@ -358,7 +461,23 @@ public void notifyCheckpointAborted(long checkpointId)
throws Exception {
keyedStateBackend.getKeyContext(),
stateChangelogWriter,
meta);
-return stateFactory.create(state, kvStateChangeLogger);
+return stateFactory.create(
+state,
+kvStateChangeLogger,
+keyedStateBackend /* pass the nested backend as key context so
that it get key updates on recovery*/);
+}
+
+private void completeRestore(Collection
stateHandles) {
+if (!stateHandles.isEmpty()) {
+synchronized (materialized) { // ensure visibility
Review comment:
IIUC, the ```buildSnapshotResult ``` runs in synchronized stage of a
checkpoint, which is already protected by the checkpoint lock.
However, since the ```materialized``` may change later upon materialization,
I do agree we add some protection here. According to the JLS
(https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5):
> An unlock on a monitor happens-before every subsequent lock on that
monitor.
I think we should do ```synchronized (materialized)``` in
```buildSnapshotResult ``` as well to ensure the _happens-before_ relationship.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16227: [FLINK-23041][streaming] Added new well defined checkpoint configuration aligned-checkpoint-timeout instead of alignment-timeout
flinkbot edited a comment on pull request #16227: URL: https://github.com/apache/flink/pull/16227#issuecomment-865242134 ## CI report: * 058af60e4592de5e83d7ba6a4165a169d986f409 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19330) * ac27568d7b4dcb40ab08da6ad5c8eb4f799d5b12 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19354) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-23075) Python API for enabling ChangelogStateBackend
[ https://issues.apache.org/jira/browse/FLINK-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-23075. --- Resolution: Fixed Merged to master via ca9b1fcdd3f2d1bc92b4b4a566fe4b084674f05e > Python API for enabling ChangelogStateBackend > - > > Key: FLINK-23075 > URL: https://issues.apache.org/jira/browse/FLINK-23075 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Affects Versions: 1.14.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > After FLINK-22678, two APIs ```enableChangelogStateBackend``` and > ```isChangelogStateBackendEnabled``` have been added. The corresponding > interfaces should be added to python API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #16241: [FLINK-23075][python] API for enabling ChangelogStateBackend
dianfu closed pull request #16241: URL: https://github.com/apache/flink/pull/16241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16227: [FLINK-23041][streaming] Added new well defined checkpoint configuration aligned-checkpoint-timeout instead of alignment-timeout
flinkbot edited a comment on pull request #16227: URL: https://github.com/apache/flink/pull/16227#issuecomment-865242134 ## CI report: * 058af60e4592de5e83d7ba6a4165a169d986f409 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19330) * ac27568d7b4dcb40ab08da6ad5c8eb4f799d5b12 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16163: [FLINK-22994][Table SQL / Planner] improve the performace of invoking…
flinkbot edited a comment on pull request #16163: URL: https://github.com/apache/flink/pull/16163#issuecomment-861284972 ## CI report: * 00f7cd34fc503db8cc116dd201bcc7f54a4b6f20 UNKNOWN * da303c24e513b0678794526c6351003b6e76cb6d UNKNOWN * 4133ee287c23af5da472db1098c9d7ab51b81cb9 UNKNOWN * 5d2e2832a0a36194e9530a6b70386e80e1f9f0ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19314) * dcb9483bb9ce418fa5470f52167ac84012db3dc4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19346) * 7382053637c1c2532527920e654538720f6c0754 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19353) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Zakelly commented on a change in pull request #15420: [FLINK-21356] Implement incremental checkpointing and recovery using state changelog
Zakelly commented on a change in pull request #15420:
URL: https://github.com/apache/flink/pull/15420#discussion_r656756445
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/inmemory/InMemoryStateChangelogWriter.java
##
@@ -61,20 +64,26 @@ public SequenceNumber lastAppendedSequenceNumber() {
}
Review comment:
It makes sense.
However, it makes me a little confused that the sequence number for records
in ```InMemoryStateChangelogWriter ``` actually starts from ```1L```, not the
```SequenceNumber.FIRST```. The word FIRST here means an initialized value
instead of a valid beginning, right? If so, it does not bother me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Zakelly commented on a change in pull request #15420: [FLINK-21356] Implement incremental checkpointing and recovery using state changelog
Zakelly commented on a change in pull request #15420:
URL: https://github.com/apache/flink/pull/15420#discussion_r656756445
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/inmemory/InMemoryStateChangelogWriter.java
##
@@ -61,20 +64,26 @@ public SequenceNumber lastAppendedSequenceNumber() {
}
Review comment:
It makes sense.
However, it makes me a little confused that sequence number for records in
```InMemoryStateChangelogWriter ``` actually starts from ```1L```, not the
```SequenceNumber.FIRST```. The word FIRST here means an initialized value
instead of a valid beginning, right? If so, it does not bother me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16247: [FLINK-23001][build] Fix missing Scala suffix for 'flink-formats/flink-avro-glue-schema-registry'
flinkbot edited a comment on pull request #16247: URL: https://github.com/apache/flink/pull/16247#issuecomment-866323741 ## CI report: * 3a1d15af39c0d2e491b92fe8c8ff2c9ecb03a337 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19339) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16248: [hotfix] Add scala version variable in pom of flink-statebackend-changelog
flinkbot edited a comment on pull request #16248: URL: https://github.com/apache/flink/pull/16248#issuecomment-866504119 ## CI report: * 60c086732f0403cd6060fd40eca38be178999cdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19351) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16220: [FLINK-23033][python] Add the missing ObjectArrayTypeInfo
flinkbot edited a comment on pull request #16220: URL: https://github.com/apache/flink/pull/16220#issuecomment-865041649 ## CI report: * ee3f8975908e9f243bb5510d0cb09a8ddf341f7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19315) * cf469ce915e4f73fbd67d24c96d6190b113ec525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19344) * 702e873d300360fe3c6a74a6c5aa9af01152cf9c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19350) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Zakelly commented on pull request #16241: [FLINK-23075][python] API for enabling ChangelogStateBackend
Zakelly commented on pull request #16241: URL: https://github.com/apache/flink/pull/16241#issuecomment-866515572 @dianfu Would you please to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16154: [FLINK-22982][python] Fix the wrong matching in PythonMapMergeRule
flinkbot edited a comment on pull request #16154: URL: https://github.com/apache/flink/pull/16154#issuecomment-861162116 ## CI report: * eecf2f97b39793896e1b56ea86aa5f6933dde515 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18972) * 6204f284bb22477b60593a7533b6414a07191e14 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19349) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-23107) Separate deduplicate rank from rank functions
Jingsong Lee created FLINK-23107: Summary: Separate deduplicate rank from rank functions Key: FLINK-23107 URL: https://issues.apache.org/jira/browse/FLINK-23107 Project: Flink Issue Type: Improvement Components: Table SQL / Runtime Reporter: Jingsong Lee Fix For: 1.14.0 SELECT * FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY d ORDER BY e DESC) AS rownum from T) WHERE rownum=1 Actually above sql is a deduplicate rank instead of a normal rank. We should separate the implementation for optimize the deduplicate rank and reduce bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23106) RetractableTopNFunction should send retract first and then send insert
[
https://issues.apache.org/jira/browse/FLINK-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-23106:
-
Description:
Case:
Source1 ---ChangeLog--->
Source2 -> Rank(rownum=1) -> Join(key is partitionKey) -> Other_Rank(partition
key is the pk of change log)
Then will be {{throw new RuntimeException( "Can not retract a non-existent
record. This should never happen.")}} in RetractableTopNFunction.
In this case, the downstream will produce the wrong result, even the wrong
retraction (retract message that does not exist)
* Rank output partition key + rownum , the rank here has row_ rownum=1
* In the downstream join judgment rank output, rownum is 1, so assuming
partition key is PK, do PK based optimization
* Because select rownum, rank thinks that the PK of downstream data must be
partition key + rownum, the retraction message of partition key is out of
order, leading to problems in downstream PK based optimization
Fix: there is a problem with the rank implementation. It should not result in
disordered retraction messages of partition key.
Bypass: do not select rownum of rank output
was:
Case:
Source1 ---ChangeLog->
Source2 -> Rank(rownum=1) -> Join(key is partitionKey) -> Other_Rank(partition
key is the pk of change log)
Then will be {{throw new RuntimeException( "Can not retract a non-existent
record. This should never happen.")}} in RetractableTopNFunction.
In this case, the downstream will produce the wrong result, even the wrong
retraction (retract message that does not exist)
* Rank output partition key + rownum , the rank here has row_ rownum=1
* In the downstream join judgment rank output, rownum is 1, so assuming
partition key is PK, do PK based optimization
* Because select rownum, rank thinks that the PK of downstream data must be
partition key + rownum, the retraction message of partition key is out of
order, leading to problems in downstream PK based optimization
Fix: there is a problem with the rank implementation. It should not result in
disordered retraction messages of partition key.
Bypass: do not select rownum of rank output
> RetractableTopNFunction should send retract first and then send insert
> --
>
> Key: FLINK-23106
> URL: https://issues.apache.org/jira/browse/FLINK-23106
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.14.0
>
>
> Case:
> Source1 ---ChangeLog--->
> Source2 -> Rank(rownum=1) -> Join(key is partitionKey) ->
> Other_Rank(partition key is the pk of change log)
> Then will be {{throw new RuntimeException( "Can not retract a non-existent
> record. This should never happen.")}} in RetractableTopNFunction.
>
> In this case, the downstream will produce the wrong result, even the wrong
> retraction (retract message that does not exist)
> * Rank output partition key + rownum , the rank here has row_ rownum=1
> * In the downstream join judgment rank output, rownum is 1, so assuming
> partition key is PK, do PK based optimization
> * Because select rownum, rank thinks that the PK of downstream data must be
> partition key + rownum, the retraction message of partition key is out of
> order, leading to problems in downstream PK based optimization
> Fix: there is a problem with the rank implementation. It should not result in
> disordered retraction messages of partition key.
> Bypass: do not select rownum of rank output
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23106) RetractableTopNFunction should send retract first and then send insert
[
https://issues.apache.org/jira/browse/FLINK-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-23106:
-
Description:
Case:
Source1 ---ChangeLog->
Source2 -> Rank(rownum=1) -> Join(key is partitionKey) -> Other_Rank(partition
key is the pk of change log)
Then will be {{throw new RuntimeException( "Can not retract a non-existent
record. This should never happen.")}} in RetractableTopNFunction.
In this case, the downstream will produce the wrong result, even the wrong
retraction (retract message that does not exist)
* Rank output partition key + rownum , the rank here has row_ rownum=1
* In the downstream join judgment rank output, rownum is 1, so assuming
partition key is PK, do PK based optimization
* Because select rownum, rank thinks that the PK of downstream data must be
partition key + rownum, the retraction message of partition key is out of
order, leading to problems in downstream PK based optimization
Fix: there is a problem with the rank implementation. It should not result in
disordered retraction messages of partition key.
Bypass: do not select rownum of rank output
was:
Consider:
Source -> Rank(rownum=1) -> Join(key is partitionKey)
In this case, the downstream will produce the wrong result, even the wrong
retraction (retract message that does not exist)
* Rank output partition key + rownum , the rank here has row_ rownum=1
* In the downstream join judgment rank output, rownum is 1, so assuming
partition key is PK, do PK based optimization
* Because select rownum, rank thinks that the PK of downstream data must be
partition key + rownum, the retraction message of partition key is out of
order, leading to problems in downstream PK based optimization
Fix: there is a problem with the rank implementation. It should not result in
disordered retraction messages of partition key.
Bypass: do not select rownum of rank output
> RetractableTopNFunction should send retract first and then send insert
> --
>
> Key: FLINK-23106
> URL: https://issues.apache.org/jira/browse/FLINK-23106
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.14.0
>
>
> Case:
> Source1 ---ChangeLog->
> Source2 -> Rank(rownum=1) -> Join(key is partitionKey) ->
> Other_Rank(partition key is the pk of change log)
> Then will be {{throw new RuntimeException( "Can not retract a non-existent
> record. This should never happen.")}} in RetractableTopNFunction.
>
> In this case, the downstream will produce the wrong result, even the wrong
> retraction (retract message that does not exist)
> * Rank output partition key + rownum , the rank here has row_ rownum=1
> * In the downstream join judgment rank output, rownum is 1, so assuming
> partition key is PK, do PK based optimization
> * Because select rownum, rank thinks that the PK of downstream data must be
> partition key + rownum, the retraction message of partition key is out of
> order, leading to problems in downstream PK based optimization
> Fix: there is a problem with the rank implementation. It should not result in
> disordered retraction messages of partition key.
> Bypass: do not select rownum of rank output
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21938) Add documentation about how to test Python UDFs
[ https://issues.apache.org/jira/browse/FLINK-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-21938. --- Assignee: Yik San Chan Resolution: Fixed Merged to master via 7063bf7abee575b19f40187b5489a916c1b61714 > Add documentation about how to test Python UDFs > --- > > Key: FLINK-21938 > URL: https://issues.apache.org/jira/browse/FLINK-21938 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Dian Fu >Assignee: Yik San Chan >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.14.0 > > > It should be similar to the Java UDFs: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-user-defined-functions -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #15360: [FLINK-21938][docs] Add how to unit test python udfs
dianfu closed pull request #15360: URL: https://github.com/apache/flink/pull/15360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-23106) RetractableTopNFunction should send retract first and then send insert
Jingsong Lee created FLINK-23106: Summary: RetractableTopNFunction should send retract first and then send insert Key: FLINK-23106 URL: https://issues.apache.org/jira/browse/FLINK-23106 Project: Flink Issue Type: Bug Components: Table SQL / Runtime Reporter: Jingsong Lee Fix For: 1.14.0 Consider: Source -> Rank(rownum=1) -> Join(key is partitionKey) In this case, the downstream will produce the wrong result, even the wrong retraction (retract message that does not exist) * Rank output partition key + rownum , the rank here has row_ rownum=1 * In the downstream join judgment rank output, rownum is 1, so assuming partition key is PK, do PK based optimization * Because select rownum, rank thinks that the PK of downstream data must be partition key + rownum, the retraction message of partition key is out of order, leading to problems in downstream PK based optimization Fix: there is a problem with the rank implementation. It should not result in disordered retraction messages of partition key. Bypass: do not select rownum of rank output -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18464) ClassCastException during namespace serialization for checkpoint (Heap and RocksDB)
[
https://issues.apache.org/jira/browse/FLINK-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367854#comment-17367854
]
guxiang commented on FLINK-18464:
-
I still have a question， the method `getPartitionedState` ，Why do we just
change namespace but not NamespaceSerializer？
> ClassCastException during namespace serialization for checkpoint (Heap and
> RocksDB)
> ---
>
> Key: FLINK-18464
> URL: https://issues.apache.org/jira/browse/FLINK-18464
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.9.3, 1.13.1
>Reporter: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-06-21-20-06-51-323.png,
> image-2021-06-21-20-07-30-281.png, image-2021-06-21-20-07-43-246.png,
> image-2021-06-21-20-33-39-295.png
>
>
> (see FLINK-23036 for error details with RocksDB)
>
> From
> [thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-failed-because-of-TimeWindow-cannot-be-cast-to-VoidNamespace-td36310.html]
> {quote}I'm using flink 1.9 on Mesos and I try to use my own trigger and
> evictor. The state is stored to memory.
> {quote}
>
>
> {code:java}
> input.setParallelism(processParallelism)
> .assignTimestampsAndWatermarks(new UETimeAssigner)
> .keyBy(_.key)
> .window(TumblingEventTimeWindows.of(Time.minutes(20)))
> .trigger(new MyTrigger)
> .evictor(new MyEvictor)
> .process(new MyFunction).setParallelism(aggregateParallelism)
> .addSink(kafkaSink).setParallelism(sinkParallelism)
> .name("kafka-record-sink"){code}
>
>
> {code:java}
> java.lang.Exception: Could not materialize checkpoint 1 for operator
> Window(TumblingEventTimeWindows(120), JoinTrigger, JoinEvictor,
> ScalaProcessWindowFunctionWrapper) -> Sink: kafka-record-sink (2/5).
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:1100)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:1042)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.ClassCastException:
> org.apache.flink.streaming.api.windowing.windows.TimeWindow cannot be cast to
> org.apache.flink.runtime.state.VoidNamespace
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:450)
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:47)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:1011)
>
> ... 3 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.windowing.windows.TimeWindow cannot be cast to
> org.apache.flink.runtime.state.VoidNamespace
> at
> org.apache.flink.runtime.state.VoidNamespaceSerializer.serialize(VoidNamespaceSerializer.java:32)
> at
> org.apache.flink.runtime.state.heap.CopyOnWriteStateMapSnapshot.writeState(CopyOnWriteStateMapSnapshot.java:114)
> at
> org.apache.flink.runtime.state.heap.AbstractStateTableSnapshot.writeStateInKeyGroup(AbstractStateTableSnapshot.java:121)
> at
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTableSnapshot.writeStateInKeyGroup(CopyOnWriteStateTableSnapshot.java:37)
> at
> org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:191)
> at
> org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:158)
> at
> org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:75)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:447)
>
> ... 5 more
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #16248: [hotfix] Add scala version variable in pom of flink-statebackend-changelog
flinkbot commented on pull request #16248: URL: https://github.com/apache/flink/pull/16248#issuecomment-866504119 ## CI report: * 60c086732f0403cd6060fd40eca38be178999cdd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16220: [FLINK-23033][python] Add the missing ObjectArrayTypeInfo
flinkbot edited a comment on pull request #16220: URL: https://github.com/apache/flink/pull/16220#issuecomment-865041649 ## CI report: * ee3f8975908e9f243bb5510d0cb09a8ddf341f7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19315) * cf469ce915e4f73fbd67d24c96d6190b113ec525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19344) * 702e873d300360fe3c6a74a6c5aa9af01152cf9c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16154: [FLINK-22982][python] Fix the wrong matching in PythonMapMergeRule
flinkbot edited a comment on pull request #16154: URL: https://github.com/apache/flink/pull/16154#issuecomment-861162116 ## CI report: * eecf2f97b39793896e1b56ea86aa5f6933dde515 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18972) * 6204f284bb22477b60593a7533b6414a07191e14 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16163: [FLINK-22994][Table SQL / Planner] improve the performace of invoking…
flinkbot edited a comment on pull request #16163: URL: https://github.com/apache/flink/pull/16163#issuecomment-861284972 ## CI report: * 00f7cd34fc503db8cc116dd201bcc7f54a4b6f20 UNKNOWN * da303c24e513b0678794526c6351003b6e76cb6d UNKNOWN * 4133ee287c23af5da472db1098c9d7ab51b81cb9 UNKNOWN * 5d2e2832a0a36194e9530a6b70386e80e1f9f0ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19314) * dcb9483bb9ce418fa5470f52167ac84012db3dc4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19346) * 7382053637c1c2532527920e654538720f6c0754 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22982) java.lang.ClassCastException when using Python UDF
[
https://issues.apache.org/jira/browse/FLINK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-22982:
Affects Version/s: (was: 1.13.1)
1.13.0
> java.lang.ClassCastException when using Python UDF
> --
>
> Key: FLINK-22982
> URL: https://issues.apache.org/jira/browse/FLINK-22982
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.0
>Reporter: Maciej Bryński
>Priority: Critical
> Labels: pull-request-available, stale-critical
> Fix For: 1.14.0, 1.13.2
>
>
> Hi,
> I'm trying to use Python UDF with logical condition as argument.
>
> {code:java}
> log = logging.getLogger()
> @udf(result_type=DataTypes.BOOLEAN())
> def trace(message, condition):
> if condition:
> log.warn(message)
> return condition
> table_env.create_temporary_function('trace', trace)
> table_env.execute_sql("""
> CREATE TABLE datagen (
> n int
> ) WITH (
> 'connector' = 'datagen',
> 'number-of-rows' = '10'
> )
> """)
> result = table_env.sql_query("""
> SELECT *
> FROM datagen
> WHERE trace(n, n < 0)
> """)
> for r in result.execute().collect():
> print(r){code}
>
> As a result I'm getting exception:
> {code:java}
> Py4JJavaError: An error occurred while calling o135.execute.
> : java.lang.ClassCastException: class org.apache.calcite.rex.RexInputRef
> cannot be cast to class org.apache.calcite.rex.RexCall
> (org.apache.calcite.rex.RexInputRef and org.apache.calcite.rex.RexCall are in
> unnamed module of loader 'app')
> at
> org.apache.flink.table.planner.plan.rules.logical.PythonMapMergeRule.matches(PythonMapMergeRule.java:70)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:538)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #16248: [hotfix] Add scala version variable in pom of flink-statebackend-changelog
flinkbot commented on pull request #16248: URL: https://github.com/apache/flink/pull/16248#issuecomment-866499564 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 60c086732f0403cd6060fd40eca38be178999cdd (Wed Jun 23 03:47:09 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Zakelly opened a new pull request #16248: [hotfix] Add scala version variable in pom of flink-statebackend-changelog
Zakelly opened a new pull request #16248: URL: https://github.com/apache/flink/pull/16248 This PR fix the error in pom of flink-statebackend-changelog, which is introduced by [641c31](https://github.com/apache/flink/commit/641c31e92fd8ff3702d2ac3510a63b0653802a2e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #16242: [FLINK-23025][upsert-kafka] Fix upsert-kafka produce duplicates when …
wuchong commented on a change in pull request #16242:
URL: https://github.com/apache/flink/pull/16242#discussion_r656734888
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/BufferedUpsertSinkFunctionTest.java
##
@@ -256,17 +273,17 @@ private BufferedUpsertSinkFunction
createBufferedSink(MockedSinkFunction sinkFun
new int[] {keyIndices},
typeInformation,
BUFFER_FLUSH_MODE);
+
bufferedSinkFunction.getRuntimeContext().getExecutionConfig().enableObjectReuse();
Review comment:
You already set object reuse in the underlying sink function, so don't
need to set here again.
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/BufferedUpsertSinkFunctionTest.java
##
@@ -129,17 +140,23 @@
TimestampData.fromInstant(Instant.parse("2021-03-30T21:00:00Z")))
};
+private final boolean enableObjectReuse;
+
+public BufferedUpsertSinkFunctionTest(boolean enableObjectReuse) {
+this.enableObjectReuse = enableObjectReuse;
+}
+
@Test
public void testWriteData() throws Exception {
-MockedSinkFunction sinkFunction = new MockedSinkFunction();
+MockedSinkFunction sinkFunction = new
MockedSinkFunction(enableObjectReuse);
Review comment:
You can make `MockedSinkFunction` non-static to avoid passing
`enableObjectReuse` here and there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-23092) Built-in UDAFs could not be mixed use with Python UDAF in group window
[ https://issues.apache.org/jira/browse/FLINK-23092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo resolved FLINK-23092. -- Resolution: Fixed Merged into master via b5d6a4349605048776c7765b1522ece73895b483 Merged into release-1.13 via 4140455083a0bb0c9240ceb1d2aac6d2024c097d > Built-in UDAFs could not be mixed use with Python UDAF in group window > -- > > Key: FLINK-23092 > URL: https://issues.apache.org/jira/browse/FLINK-23092 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.13.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367832#comment-17367832
]
shizhengchao edited comment on FLINK-23096 at 6/23/21, 3:29 AM:
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
public static Path getHDFSSessionPath(Configuration conf) {
SessionState ss = SessionState.get();
if (ss == null) {
String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
Preconditions.checkNotNull(sessionPathString,
"Conf non-local session path expected to be non-null");
return new Path(sessionPathString);
}
Preconditions.checkNotNull(ss.hdfsSessionPath,
"Non-local session path expected to be non-null");
return ss.hdfsSessionPath;
}
}{code}
beacuse the session is closed，so SessionState.get() is null.
was (Author: tinny):
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
public static Path getHDFSSessionPath(Configuration conf) {
SessionState ss = SessionState.get();
if (ss == null) {
String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
Preconditions.checkNotNull(sessionPathString,
"Conf non-local session path expected to be non-null");
return new Path(sessionPathString);
}
Preconditions.checkNotNull(ss.hdfsSessionPath,
"Non-local session path expected to be non-null");
return ss.hdfsSessionPath;
}
}{code}
beacuse the session is closed，so SessionState.get() will be cause an NPE.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at
[jira] [Updated] (FLINK-23092) Built-in UDAFs could not be mixed use with Python UDAF in group window
[ https://issues.apache.org/jira/browse/FLINK-23092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-23092: Affects Version/s: (was: 1.13.1) 1.13.0 > Built-in UDAFs could not be mixed use with Python UDAF in group window > -- > > Key: FLINK-23092 > URL: https://issues.apache.org/jira/browse/FLINK-23092 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.13.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23092) Built-in UDAFs could not be mixed use with Python UDAF in group window
[ https://issues.apache.org/jira/browse/FLINK-23092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-23092: Summary: Built-in UDAFs could not be mixed use with Python UDAF in group window (was: Fix built-in functions are unsupported in Python Group Window UDAF) > Built-in UDAFs could not be mixed use with Python UDAF in group window > -- > > Key: FLINK-23092 > URL: https://issues.apache.org/jira/browse/FLINK-23092 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.13.1 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16163: [FLINK-22994][Table SQL / Planner] improve the performace of invoking…
flinkbot edited a comment on pull request #16163: URL: https://github.com/apache/flink/pull/16163#issuecomment-861284972 ## CI report: * 00f7cd34fc503db8cc116dd201bcc7f54a4b6f20 UNKNOWN * da303c24e513b0678794526c6351003b6e76cb6d UNKNOWN * 4133ee287c23af5da472db1098c9d7ab51b81cb9 UNKNOWN * 5d2e2832a0a36194e9530a6b70386e80e1f9f0ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19314) * dcb9483bb9ce418fa5470f52167ac84012db3dc4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo closed pull request #16240: [FLINK-23092][python] Fix the issue that built-in UDAF could not be mixed use with Python UDAF in group window
HuangXingBo closed pull request #16240: URL: https://github.com/apache/flink/pull/16240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-22643) Too many TCP connections among TaskManagers for large scale jobs
[
https://issues.apache.org/jira/browse/FLINK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367833#comment-17367833
]
Zhilong Hong edited comment on FLINK-22643 at 6/23/21, 3:23 AM:
Only tasks that belong to the same JobVertex on the same TaskManager will reuse
the same TCP connections. As slot sharing is enabled, there will be many tasks
that belong to different JobVertices on the same TaskManager. They cannot share
the TCP connections.
Thus we propose to makes all tasks on the same TaskManager reuse one TCP
connection. This will decrease the number of TCP connections, especially for
the job with many JobVertices.
was (Author: thesharing):
Only tasks that belong to the same JobVertex on the same TaskManager will
re-use the same TCP connections. Thus we propose to makes all tasks on the same
TaskManager re-use one TCP connection. This will decrease the number of TCP
connections, especially for the job with many JobVertices.
> Too many TCP connections among TaskManagers for large scale jobs
>
>
> Key: FLINK-22643
> URL: https://issues.apache.org/jira/browse/FLINK-22643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: stale-major
> Fix For: 1.14.0
>
>
> For the large scale jobs, there will be too many TCP connections among
> TaskManagers. Let's take an example.
> For a streaming job with 20 JobVertices, each JobVertex has 500 parallelism.
> We divide the vertices into 5 slot sharing groups. Each TaskManager has 5
> slots. Thus there will be 400 taskmanagers in this job. Let's assume that job
> runs on a cluster with 20 machines.
> If all the job edges are all-to-all edges, there will be 19 * 20 * 399 * 2 =
> 303,240 TCP connections for each machine. If we run several jobs on this
> cluster, the TCP connections may exceed the maximum limit of linux, which is
> 1,048,576. This will stop the TaskManagers from creating new TCP connections
> and cause task failovers.
> As we run our production jobs on a K8S cluster, the job always failover due
> to exceptions related to network, such as {{Sending the partition request to
> 'null' failed}}, and etc.
> We think that we can decrease the number of connections by letting tasks
> reuse the same connection. We implemented a POC that makes all tasks on the
> same TaskManager reuse one TCP connection. For the example job we mentioned
> above, the number of connections will decrease from 303,240 to 15960. With
> the POC, the frequency of meeting exceptions related to network in our
> production jobs drops significantly.
> The POC is illustrated in:
> https://github.com/wsry/flink/commit/bf1c09e80450f40d018a1d1d4fe3dfd2de777fdc
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-22953) Using Types.LIST(Types.STRING()) in datastream result in crash
[
https://issues.apache.org/jira/browse/FLINK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-22953.
---
Resolution: Duplicate
> Using Types.LIST(Types.STRING()) in datastream result in crash
> --
>
> Key: FLINK-22953
> URL: https://issues.apache.org/jira/browse/FLINK-22953
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.1
> Environment: python==3.7.5
> flink==1.13.1
>Reporter: awayne
>Priority: Critical
> Labels: stale-critical
>
> In our business, we need to deserialize JSON like this:
> {code:java}
> {
> ...,
> "texts": ["a", "b", "c"],
> ...
> }{code}
> number of strings in list isn't fixed.
> After define Row type using Types.LIST(Types.STRING()), job will crash:
> {code:java}
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute return
> JobExecutionResult(self._j_stream_execution_environment.execute(j_stream_graph))
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/java_gateway.py", line
> 1286, in __call__ answer, self.gateway_client, self.target_id, self.name)
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/util/exceptions.py",
> line 146, in deco return f(*a, **kw) File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/protocol.py", line
> 328, in get_return_value format(target_id, ".", name),
> value)py4j.protocol.Py4JJavaError: An error occurred while calling
> o31.execute.: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed. at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
> at akka.dispatch.OnComplete.internal(Future.scala:264) at
> akka.dispatch.OnComplete.internal(Future.scala:261) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572) at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436) at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
>
[jira] [Commented] (FLINK-22953) Using Types.LIST(Types.STRING()) in datastream result in crash
[
https://issues.apache.org/jira/browse/FLINK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367836#comment-17367836
]
Dian Fu commented on FLINK-22953:
-
[~awayne] Thanks for the confirmation. Then I will close this issue as
duplicate as there is already a PR in FLINK-23033.
Regarding to the documentation, I agree with you that the documentation is not
complete. I have already updated the documentation about the data types
supported in Python DataStream API in FLINK-23033.
> Using Types.LIST(Types.STRING()) in datastream result in crash
> --
>
> Key: FLINK-22953
> URL: https://issues.apache.org/jira/browse/FLINK-22953
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.1
> Environment: python==3.7.5
> flink==1.13.1
>Reporter: awayne
>Priority: Critical
> Labels: stale-critical
>
> In our business, we need to deserialize JSON like this:
> {code:java}
> {
> ...,
> "texts": ["a", "b", "c"],
> ...
> }{code}
> number of strings in list isn't fixed.
> After define Row type using Types.LIST(Types.STRING()), job will crash:
> {code:java}
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute return
> JobExecutionResult(self._j_stream_execution_environment.execute(j_stream_graph))
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/java_gateway.py", line
> 1286, in __call__ answer, self.gateway_client, self.target_id, self.name)
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/util/exceptions.py",
> line 146, in deco return f(*a, **kw) File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/protocol.py", line
> 328, in get_return_value format(target_id, ".", name),
> value)py4j.protocol.Py4JJavaError: An error occurred while calling
> o31.execute.: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed. at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
> at akka.dispatch.OnComplete.internal(Future.scala:264) at
> akka.dispatch.OnComplete.internal(Future.scala:261) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572) at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436) at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
[jira] [Updated] (FLINK-22915) Update Flink ML library to support Estimator/Transformer DAG and online learning
[ https://issues.apache.org/jira/browse/FLINK-22915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-22915: - Description: The existing Flink ML library allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each Estimator/Transformer has one input and one output. The following use-cases are not supported yet. And we propose FLIP-173 [1] to address these use-cases. 1) Express an Estimator/Transformer that has multiple inputs/outputs. For example, some graph embedding algorithms need to take two tables as inputs. These two tables represent nodes and edges of the graph respectively. This logic can be expressed as an Estimator with 2 input tables. And some workflow may need to split 1 table into 2 tables, and use these tables for training and validation respectively. This logic can be expressed by a Transformer with 1 input table and 2 output tables. 2) Compose a directed-acyclic-graph Estimator/Transformer into an Estimator/Transformer. For example, the workflow may involve the join of 2 tables, where each table could be generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer. 3) Online learning where a long-running instance Transformer needs to be updated by the latest model data generated by another long-running instance of Estimator. In this scenario, we need to allow the Estimator to be run on a different machine than the Transformer. So that Estimator could consume sufficient computation resource in a cluster while the Transformer could be deployed on edge devices. In addition to addressing the above use-cases, we also propose a few more changes to simplify the class hierarchy and improve API usability. The existing Flink ML library has the following usability issues: 4) The Model interface does not provide any added value (given that we already have Transformer). The added class hierarchy complexity is not justified. 5) fit/transform API requires users to explicitly provide the TableEnvironment, where the TableEnvironment could be retrieved from the Table instance given to the fit/transform. 6) A Pipeline is both a Transformer and an Estimator. The experience of using Pipeline is therefore different from the experience of using Estimator (with the needFit API). 7) There is no API provided by the Estimator/Transformer interface to validate the schema consistency of a Pipeline. Users would have to instantiate Tables (with I/O logics) and run fit/transform to know whether the stages in the Pipeline are compatible with each other. was: The existing Flink ML API allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each Estimator/Transformer has one input and one output. The following use-cases are not supported yet. And we propose FLIP-173 [1] to address these use-cases. 1) Express an Estimator/Transformer that has multiple inputs/outputs. For example, some graph embedding algorithms need to take two tables as inputs. These two tables represent nodes and edges of the graph respectively. This logic can be expressed as an Estimator with 2 input tables. And some workflow may need to split 1 table into 2 tables, and use these tables for training and validation respectively. This logic can be expressed by a Transformer with 1 input table and 2 output tables. 2) Compose a directed-acyclic-graph Estimator/Transformer into an Estimator/Transformer. For example, the workflow may involve the join of two tables, where each table are generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer. 3) Online learning where a long-running instance Transformer needs to be updated by the latest model data generated by another long-running instance of Estimator. In this scenario, we need to allow the Estimator to be run on a different machine than the Transformer. So that Estimator could consume sufficient computation resource in a cluster while the Transformer could be deployed on edge devices. > Update Flink ML library to support Estimator/Transformer DAG and online > learning > > > Key: FLINK-22915 > URL: https://issues.apache.org/jira/browse/FLINK-22915 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Dong Lin >Priority: Major > > The existing Flink ML library allows users to compose an > Estimator/Transformer from a pipeline (i.e. linear sequence) of > Estimator/Transformer, and each Estimator/Transformer has one input and one > output. > The following use-cases are not supported yet. And we propose FLIP-173 [1] to > address
[jira] [Commented] (FLINK-22953) Using Types.LIST(Types.STRING()) in datastream result in crash
[
https://issues.apache.org/jira/browse/FLINK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367835#comment-17367835
]
awayne commented on FLINK-22953:
[~dian.fu] It's the same problem.
btw, I just notice that using Types.BASIC_ARRAY(Types.STRING()) can meet our
demand，but docs dosen't mention it:
[https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/python/datastream/data_types/#supported-data-types]
it need to be updated.
> Using Types.LIST(Types.STRING()) in datastream result in crash
> --
>
> Key: FLINK-22953
> URL: https://issues.apache.org/jira/browse/FLINK-22953
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.1
> Environment: python==3.7.5
> flink==1.13.1
>Reporter: awayne
>Priority: Critical
> Labels: stale-critical
>
> In our business, we need to deserialize JSON like this:
> {code:java}
> {
> ...,
> "texts": ["a", "b", "c"],
> ...
> }{code}
> number of strings in list isn't fixed.
> After define Row type using Types.LIST(Types.STRING()), job will crash:
> {code:java}
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/datastream/stream_execution_environment.py",
> line 645, in execute return
> JobExecutionResult(self._j_stream_execution_environment.execute(j_stream_graph))
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/java_gateway.py", line
> 1286, in __call__ answer, self.gateway_client, self.target_id, self.name)
> File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/pyflink/util/exceptions.py",
> line 146, in deco return f(*a, **kw) File
> "/home/ubuntu/pyflenv/lib/python3.7/site-packages/py4j/protocol.py", line
> 328, in get_return_value format(target_id, ".", name),
> value)py4j.protocol.Py4JJavaError: An error occurred while calling
> o31.execute.: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed. at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
> at akka.dispatch.OnComplete.internal(Future.scala:264) at
> akka.dispatch.OnComplete.internal(Future.scala:261) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572) at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436) at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435) at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at
>
[jira] [Commented] (FLINK-23104) flink-statebackend-changelog does not build with scala 2.12
[ https://issues.apache.org/jira/browse/FLINK-23104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367834#comment-17367834 ] Zakelly Lan commented on FLINK-23104: - [~xintongsong] Yeah, my fault. I'll fix it. Thanks > flink-statebackend-changelog does not build with scala 2.12 > --- > > Key: FLINK-23104 > URL: https://issues.apache.org/jira/browse/FLINK-23104 > Project: Flink > Issue Type: Bug > Components: Build System, Runtime / State Backends >Affects Versions: 1.14.0 >Reporter: Xintong Song >Priority: Blocker > Fix For: 1.14.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065=4868 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367832#comment-17367832
]
shizhengchao edited comment on FLINK-23096 at 6/23/21, 3:10 AM:
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
public static Path getHDFSSessionPath(Configuration conf) {
SessionState ss = SessionState.get();
if (ss == null) {
String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
Preconditions.checkNotNull(sessionPathString,
"Conf non-local session path expected to be non-null");
return new Path(sessionPathString);
}
Preconditions.checkNotNull(ss.hdfsSessionPath,
"Non-local session path expected to be non-null");
return ss.hdfsSessionPath;
}
}{code}
beacuse the session is closed，so SessionState.get() will be cause an NPE.
was (Author: tinny):
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}{code}
beacuse the session is closed，so SessionState.getHDFSSessionPath(hiveConf) will
be cause an NPE.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf
[jira] [Comment Edited] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367832#comment-17367832
]
shizhengchao edited comment on FLINK-23096 at 6/23/21, 3:09 AM:
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符
private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}{code}
beacuse the session is closed，so SessionState.getHDFSSessionPath(hiveConf) will
be cause an NPE.
was (Author: tinny):
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}{code}
beacuse the session is closed，so SessionState.getHDFSSessionPath(hiveConf) will
be cause an NPE.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
>
[jira] [Commented] (FLINK-22643) Too many TCP connections among TaskManagers for large scale jobs
[
https://issues.apache.org/jira/browse/FLINK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367833#comment-17367833
]
Zhilong Hong commented on FLINK-22643:
--
Only tasks that belong to the same JobVertex on the same TaskManager will
re-use the same TCP connections. Thus we propose to makes all tasks on the same
TaskManager re-use one TCP connection. This will decrease the number of TCP
connections, especially for the job with many JobVertices.
> Too many TCP connections among TaskManagers for large scale jobs
>
>
> Key: FLINK-22643
> URL: https://issues.apache.org/jira/browse/FLINK-22643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: stale-major
> Fix For: 1.14.0
>
>
> For the large scale jobs, there will be too many TCP connections among
> TaskManagers. Let's take an example.
> For a streaming job with 20 JobVertices, each JobVertex has 500 parallelism.
> We divide the vertices into 5 slot sharing groups. Each TaskManager has 5
> slots. Thus there will be 400 taskmanagers in this job. Let's assume that job
> runs on a cluster with 20 machines.
> If all the job edges are all-to-all edges, there will be 19 * 20 * 399 * 2 =
> 303,240 TCP connections for each machine. If we run several jobs on this
> cluster, the TCP connections may exceed the maximum limit of linux, which is
> 1,048,576. This will stop the TaskManagers from creating new TCP connections
> and cause task failovers.
> As we run our production jobs on a K8S cluster, the job always failover due
> to exceptions related to network, such as {{Sending the partition request to
> 'null' failed}}, and etc.
> We think that we can decrease the number of connections by letting tasks
> reuse the same connection. We implemented a POC that makes all tasks on the
> same TaskManager reuse one TCP connection. For the example job we mentioned
> above, the number of connections will decrease from 303,240 to 15960. With
> the POC, the frequency of meeting exceptions related to network in our
> production jobs drops significantly.
> The POC is illustrated in:
> https://github.com/wsry/flink/commit/bf1c09e80450f40d018a1d1d4fe3dfd2de777fdc
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367832#comment-17367832
]
shizhengchao edited comment on FLINK-23096 at 6/23/21, 3:07 AM:
[~Leonard Xu] [~lirui]
{code:java}
//代码占位符private void clearSessionState(HiveConf hiveConf) {
SessionState sessionState = SessionState.get();
if (sessionState != null) {
try {
sessionState.close();
List toDelete = new ArrayList<>();
toDelete.add(SessionState.getHDFSSessionPath(hiveConf));
toDelete.add(SessionState.getLocalSessionPath(hiveConf));
for (Path path : toDelete) {
FileSystem fs = path.getFileSystem(hiveConf);
fs.delete(path, true);
}
} catch (IOException e) {
LOG.warn("Error closing SessionState", e);
}
}
}{code}
beacuse the session is closed，so SessionState.getHDFSSessionPath(hiveConf) will
be cause an NPE.
was (Author: tinny):
[~Leonard Xu] [~lirui]
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367832#comment-17367832
]
shizhengchao commented on FLINK-23096:
--
[~Leonard Xu] [~lirui]
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16246: [BP-1.13][FLINK-23073][formats / CSV] Fix space handling in Row CSV timestamp parser
flinkbot edited a comment on pull request #16246: URL: https://github.com/apache/flink/pull/16246#issuecomment-866294192 ## CI report: * a0006b2f922fe4e6223cde71a2dddf2ce20189d0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19338) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16233: [FLINK-22085][tests] Add debug code to expose the sink status for KafkaSourceLegacyITCase
flinkbot edited a comment on pull request #16233: URL: https://github.com/apache/flink/pull/16233#issuecomment-865644244 ## CI report: * 4c8e3dd9ae5130d58fe537c8b60e0aa7f6060b1e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19345) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19303) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22927) Exception on JobClient.get_job_status().result()
[
https://issues.apache.org/jira/browse/FLINK-22927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-22927:
Fix Version/s: 1.13.2
> Exception on JobClient.get_job_status().result()
>
>
> Key: FLINK-22927
> URL: https://issues.apache.org/jira/browse/FLINK-22927
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.0
>Reporter: Maciej Bryński
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.2
>
>
> Following code finish with exception
> {code:java}
> table_env.execute_sql("""
> CREATE TABLE IF NOT EXISTS datagen (
> id INT,
> data STRING
> ) WITH (
> 'connector' = 'datagen'
> )
> table_env.execute_sql("""
> CREATE TABLE IF NOT EXISTS print (
> id INT,
> data STRING
> ) WITH (
> 'connector' = 'print'
> )
> """){code}
> {code:java}
> table_result = table_env.execute_sql("INSERT INTO print SELECT * FROM
> datagen")
> table_result.get_job_client().get_job_status().result()
> ---
> ValueErrorTraceback (most recent call last)
> ValueError: JavaObject id=o125 is not a valid JobStatus
> During handling of the above exception, another exception occurred:
> ValueErrorTraceback (most recent call last)
> in
> > 1 result.get_job_client().get_job_status().result()
> /usr/local/lib/python3.8/dist-packages/pyflink/common/completable_future.py
> in result(self)
> 76 return self._j_completable_future.get()
> 77 else:
> ---> 78 return self._py_class(self._j_completable_future.get())
> 79
> 80 def exception(self):
> /usr/lib/python3.8/enum.py in __call__(cls, value, names, module, qualname,
> type, start)
> 307 """
> 308 if names is None: # simple value lookup
> --> 309 return cls.__new__(cls, value)
> 310 # otherwise, functional API: we're creating a new Enum type
> 311 return cls._create_(value, names, module=module,
> qualname=qualname, type=type, start=start)
> /usr/lib/python3.8/enum.py in __new__(cls, value)
> 598 )
> 599 exc.__context__ = ve_exc
> --> 600 raise exc
> 601
> 602 def _generate_next_value_(name, start, count, last_values):
> /usr/lib/python3.8/enum.py in __new__(cls, value)
> 582 try:
> 583 exc = None
> --> 584 result = cls._missing_(value)
> 585 except Exception as e:
> 586 exc = e
> /usr/lib/python3.8/enum.py in _missing_(cls, value)
> 611 @classmethod
> 612 def _missing_(cls, value):
> --> 613 raise ValueError("%r is not a valid %s" % (value,
> cls.__name__))
> 614
> 615 def __repr__(self):
> ValueError: JavaObject id=o125 is not a valid JobStatus
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16233: [FLINK-22085][tests] Add debug code to expose the sink status for KafkaSourceLegacyITCase
flinkbot edited a comment on pull request #16233: URL: https://github.com/apache/flink/pull/16233#issuecomment-865644244 ## CI report: * 4c8e3dd9ae5130d58fe537c8b60e0aa7f6060b1e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19303) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19345) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-23105) YARNSessionCapacitySchedulerITCase.testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots fail
Xintong Song created FLINK-23105:
Summary:
YARNSessionCapacitySchedulerITCase.testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots
fails on azure
Key: FLINK-23105
URL: https://issues.apache.org/jira/browse/FLINK-23105
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.14.0
Reporter: Xintong Song
Fix For: 1.14.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=9c1ddabe-d186-5a2c-5fcc-f3cafb3ec699=28045
{code}
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Failed to execute job 'Windowed Join Example'.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.yarn.YarnTestBase$Runner.run(YarnTestBase.java:1144)
Caused by: org.apache.flink.util.FlinkException: Failed to execute job
'Windowed Join Example'.
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2027)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
at
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1891)
at
org.apache.flink.streaming.examples.join.WindowJoin.main(WindowJoin.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 6 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
submit JobGraph.
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$9(RestClusterClient.java:405)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:386)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:430)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at
[GitHub] [flink] flinkbot edited a comment on pull request #16220: [FLINK-23033][python] Add the missing ObjectArrayTypeInfo
flinkbot edited a comment on pull request #16220: URL: https://github.com/apache/flink/pull/16220#issuecomment-865041649 ## CI report: * ee3f8975908e9f243bb5510d0cb09a8ddf341f7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19315) * cf469ce915e4f73fbd67d24c96d6190b113ec525 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19344) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16146: [FLINK-22927][python] Fix the bug of JobStatus
flinkbot edited a comment on pull request #16146: URL: https://github.com/apache/flink/pull/16146#issuecomment-859379229 ## CI report: * 2c66d73b141988c8c2e2aa7b92fa2df763e2a06f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18970) * ce5eeaa31e866ed8950567b10ec973f85aef8d7a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19343) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22915) Update Flink ML library to support Estimator/Transformer DAG and online learning
[ https://issues.apache.org/jira/browse/FLINK-22915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-22915: - Description: The existing Flink ML API allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each Estimator/Transformer has one input and one output. The following use-cases are not supported yet. And we propose FLIP-173 [1] to address these use-cases. 1) Express an Estimator/Transformer that has multiple inputs/outputs. For example, some graph embedding algorithms need to take two tables as inputs. These two tables represent nodes and edges of the graph respectively. This logic can be expressed as an Estimator with 2 input tables. And some workflow may need to split 1 table into 2 tables, and use these tables for training and validation respectively. This logic can be expressed by a Transformer with 1 input table and 2 output tables. 2) Compose a directed-acyclic-graph Estimator/Transformer into an Estimator/Transformer. For example, the workflow may involve the join of two tables, where each table are generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer. 3) Online learning where a long-running instance Transformer needs to be updated by the latest model data generated by another long-running instance of Estimator. In this scenario, we need to allow the Estimator to be run on a different machine than the Transformer. So that Estimator could consume sufficient computation resource in a cluster while the Transformer could be deployed on edge devices. was: Currently Flink ML API allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each Estimator/Transformer has one input and one output. The following use-cases are not supported yet. And we propose FLIP-173 [1] to address these use-cases. 1) Express an Estimator/Transformer that has multiple inputs/outputs. For example, some graph embedding algorithms need to take two tables as inputs. These two tables represent nodes and edges of the graph respectively. This logic can be expressed as an Estimator with 2 input tables. And some workflow may need to split 1 table into 2 tables, and use these tables for training and validation respectively. This logic can be expressed by a Transformer with 1 input table and 2 output tables. 2) Compose a directed-acyclic-graph Estimator/Transformer into an Estimator/Transformer. For example, the workflow may involve the join of two tables, where each table are generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer. 3) Online learning where a long-running instance Transformer needs to be updated by the latest model data generated by another long-running instance of Estimator. In this scenario, we need to allow the Estimator to be run on a different machine than the Transformer. So that Estimator could consume sufficient computation resource in a cluster while the Transformer could be deployed on edge devices. > Update Flink ML library to support Estimator/Transformer DAG and online > learning > > > Key: FLINK-22915 > URL: https://issues.apache.org/jira/browse/FLINK-22915 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Dong Lin >Priority: Major > > The existing Flink ML API allows users to compose an Estimator/Transformer > from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each > Estimator/Transformer has one input and one output. > The following use-cases are not supported yet. And we propose FLIP-173 [1] to > address these use-cases. > 1) Express an Estimator/Transformer that has multiple inputs/outputs. > For example, some graph embedding algorithms need to take two tables as > inputs. These two tables represent nodes and edges of the graph respectively. > This logic can be expressed as an Estimator with 2 input tables. > And some workflow may need to split 1 table into 2 tables, and use these > tables for training and validation respectively. This logic can be expressed > by a Transformer with 1 input table and 2 output tables. > 2) Compose a directed-acyclic-graph Estimator/Transformer into an > Estimator/Transformer. > For example, the workflow may involve the join of two tables, where each > table are generated by a chain of Estimator/Transformer. The entire workflow > is therefore a DAG of Estimator/Transformer. > 3) Online learning where a long-running instance Transformer needs to be > updated by the latest model data generated by another long-running instance > of Estimator. > In this scenario,
[jira] [Updated] (FLINK-22915) Update Flink ML library to support Estimator/Transformer DAG and online learning
[ https://issues.apache.org/jira/browse/FLINK-22915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-22915: - Description: Currently Flink ML API allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer, and each Estimator/Transformer has one input and one output. The following use-cases are not supported yet. And we propose FLIP-173 [1] to address these use-cases. 1) Express an Estimator/Transformer that has multiple inputs/outputs. For example, some graph embedding algorithms need to take two tables as inputs. These two tables represent nodes and edges of the graph respectively. This logic can be expressed as an Estimator with 2 input tables. And some workflow may need to split 1 table into 2 tables, and use these tables for training and validation respectively. This logic can be expressed by a Transformer with 1 input table and 2 output tables. 2) Compose a directed-acyclic-graph Estimator/Transformer into an Estimator/Transformer. For example, the workflow may involve the join of two tables, where each table are generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer. 3) Online learning where a long-running instance Transformer needs to be updated by the latest model data generated by another long-running instance of Estimator. In this scenario, we need to allow the Estimator to be run on a different machine than the Transformer. So that Estimator could consume sufficient computation resource in a cluster while the Transformer could be deployed on edge devices. was: Currently Flink ML API allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer. We propose to extend the Flink ML API so that users can compose an Estimator/Transformer from a directed-acyclic-graph (i.e. DAG) of Estimator/Transformer. This feature is useful for the following use-cases: 1) The preprocessing workflow (shared between training and inference workflows) may involve the join of multiple tables, where the join of two tables can be expressed as a Transformer of 2 inputs and 1 output. And the preprocessing workflow could also involve the spilt operation, where the split operation has 1 input (e.g. the original table) and 2 outputs (e.g. the split of the original table). The expression of preprocessing workflow involving the join/split operation needs to be expressed as a DAG of Transformer. 2) The graph-embedding algorithm can be expressed as an Estimator, where the Estimator takes as input two tables (e.g. a node table and an edge table). The corresponding Transformer has 1 input (i.e. the node) and 1 output (i.e. the node after embedding) The expression of training workflow involving the graph-embedding Estimator needs to be expressed as a DAG of Transformer/Estimator. Summary: Update Flink ML library to support Estimator/Transformer DAG and online learning (was: Extend Flink ML API to support Estimator/Transformer DAG) > Update Flink ML library to support Estimator/Transformer DAG and online > learning > > > Key: FLINK-22915 > URL: https://issues.apache.org/jira/browse/FLINK-22915 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Dong Lin >Priority: Major > > Currently Flink ML API allows users to compose an Estimator/Transformer from > a pipeline (i.e. linear sequence) of Estimator/Transformer, and each > Estimator/Transformer has one input and one output. > The following use-cases are not supported yet. And we propose FLIP-173 [1] to > address these use-cases. > 1) Express an Estimator/Transformer that has multiple inputs/outputs. > For example, some graph embedding algorithms need to take two tables as > inputs. These two tables represent nodes and edges of the graph respectively. > This logic can be expressed as an Estimator with 2 input tables. > And some workflow may need to split 1 table into 2 tables, and use these > tables for training and validation respectively. This logic can be expressed > by a Transformer with 1 input table and 2 output tables. > 2) Compose a directed-acyclic-graph Estimator/Transformer into an > Estimator/Transformer. > For example, the workflow may involve the join of two tables, where each > table are generated by a chain of Estimator/Transformer. The entire workflow > is therefore a DAG of Estimator/Transformer. > 3) Online learning where a long-running instance Transformer needs to be > updated by the latest model data generated by another long-running instance > of Estimator. > In this scenario, we need to allow the Estimator to be run on a different > machine
[jira] [Commented] (FLINK-22382) ProcessFailureCancelingITCase.testCancelingOnProcessFailure
[
https://issues.apache.org/jira/browse/FLINK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367824#comment-17367824
]
Xintong Song commented on FLINK-22382:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=2c7d57b9-7341-5a87-c9af-2cf7cc1a37dc=4277
> ProcessFailureCancelingITCase.testCancelingOnProcessFailure
> ---
>
> Key: FLINK-22382
> URL: https://issues.apache.org/jira/browse/FLINK-22382
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16896=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=9756
> {code}
> Apr 20 18:05:14 Suppressed: java.util.concurrent.TimeoutException
> Apr 20 18:05:14 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> Apr 20 18:05:14 at
> org.apache.flink.test.recovery.ProcessFailureCancelingITCase.waitUntilAtLeastOneTaskHasBeenDeployed(ProcessFailureCancelingITCase.java:236)
> Apr 20 18:05:14 at
> org.apache.flink.test.recovery.ProcessFailureCancelingITCase.testCancelingOnProcessFailure(ProcessFailureCancelingITCase.java:193)
> Apr 20 18:05:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Apr 20 18:05:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 20 18:05:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 20 18:05:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> Apr 20 18:05:14 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 20 18:05:14 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 20 18:05:14 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 20 18:05:14 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Apr 20 18:05:14 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Apr 20 18:05:14 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Apr 20 18:05:14 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Apr 20 18:05:14 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Apr 20 18:05:14 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #15531: [FLINK-22147][connector/kafka] Refactor partition discovery logic in Kafka source enumerator
flinkbot edited a comment on pull request #15531: URL: https://github.com/apache/flink/pull/15531#issuecomment-816343442 ## CI report: * ef0489f55f950c18c3627296fd8db333a0e0d6af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18807) * 1503664338eb8cd7310e2c8a909c3599ffd72573 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19342) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20495) Elasticsearch6DynamicSinkITCase Hang
[
https://issues.apache.org/jira/browse/FLINK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367823#comment-17367823
]
Xintong Song commented on FLINK-20495:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51=11716
> Elasticsearch6DynamicSinkITCase Hang
>
>
> Key: FLINK-20495
> URL: https://issues.apache.org/jira/browse/FLINK-20495
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors /
> ElasticSearch, Tests
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10535=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20]
>
> {code:java}
> 2020-12-04T22:39:33.9748225Z [INFO] Running
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase
> 2020-12-04T22:54:51.9486410Z
> ==
> 2020-12-04T22:54:51.9488766Z Process produced no output for 900 seconds.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23104) flink-statebackend-changelog does not build with scala 2.12
[ https://issues.apache.org/jira/browse/FLINK-23104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367822#comment-17367822 ] Xintong Song commented on FLINK-23104: -- Seems related to FLINK-22678. cc [~Zakelly] > flink-statebackend-changelog does not build with scala 2.12 > --- > > Key: FLINK-23104 > URL: https://issues.apache.org/jira/browse/FLINK-23104 > Project: Flink > Issue Type: Bug > Components: Build System, Runtime / State Backends >Affects Versions: 1.14.0 >Reporter: Xintong Song >Priority: Blocker > Fix For: 1.14.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065=4868 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23104) flink-statebackend-changelog does not build with scala 2.12
Xintong Song created FLINK-23104: Summary: flink-statebackend-changelog does not build with scala 2.12 Key: FLINK-23104 URL: https://issues.apache.org/jira/browse/FLINK-23104 Project: Flink Issue Type: Bug Components: Build System, Runtime / State Backends Affects Versions: 1.14.0 Reporter: Xintong Song Fix For: 1.14.0 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065=4868 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23052) cron_snapshot_deployment_maven unstable on maven
[
https://issues.apache.org/jira/browse/FLINK-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367815#comment-17367815
]
Xintong Song commented on FLINK-23052:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> cron_snapshot_deployment_maven unstable on maven
>
>
> Key: FLINK-23052
> URL: https://issues.apache.org/jira/browse/FLINK-23052
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Reporter: Xintong Song
>Priority: Major
>
> {{cron_snapshot_deployment_maven}}, the cron build on azure that deploys
> snapshot artifacts to maven central repository, has become unstable recently.
> The failures fall into two categories.
> - Maven failed to upload/download an artifact
> - The stage overall takes too long time.
> As far as I can see, the instability starts being observed since June 18th.
> Observed instances:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19152=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19154=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19167=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19169=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19176=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19175=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19177=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21511) Flink connector elasticsearch 6.x has a bug about BulkProcessor hangs for threads deadlocked
[ https://issues.apache.org/jira/browse/FLINK-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367814#comment-17367814 ] Yangze Guo commented on FLINK-21511: ping [~AHeise] > Flink connector elasticsearch 6.x has a bug about BulkProcessor hangs for > threads deadlocked > - > > Key: FLINK-21511 > URL: https://issues.apache.org/jira/browse/FLINK-21511 > Project: Flink > Issue Type: Bug > Components: Connectors / ElasticSearch >Affects Versions: 1.10.3, 1.11.3, 1.12.1 >Reporter: zhangmeng >Priority: Major > Labels: auto-unassigned, pull-request-available > > We use flink1.10, flink elasticsearch connector 6.x to write elasticsearch. A > total of 50 tasks running a weeks. There were more than 30 tasks that no > longer wrote data. Investigation found that there was a deadlock bug in the > current version of elasticsearch. and fixed on high version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22382) ProcessFailureCancelingITCase.testCancelingOnProcessFailure
[
https://issues.apache.org/jira/browse/FLINK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367813#comment-17367813
]
Xintong Song commented on FLINK-22382:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19335=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=9592
> ProcessFailureCancelingITCase.testCancelingOnProcessFailure
> ---
>
> Key: FLINK-22382
> URL: https://issues.apache.org/jira/browse/FLINK-22382
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16896=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=9756
> {code}
> Apr 20 18:05:14 Suppressed: java.util.concurrent.TimeoutException
> Apr 20 18:05:14 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> Apr 20 18:05:14 at
> org.apache.flink.test.recovery.ProcessFailureCancelingITCase.waitUntilAtLeastOneTaskHasBeenDeployed(ProcessFailureCancelingITCase.java:236)
> Apr 20 18:05:14 at
> org.apache.flink.test.recovery.ProcessFailureCancelingITCase.testCancelingOnProcessFailure(ProcessFailureCancelingITCase.java:193)
> Apr 20 18:05:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Apr 20 18:05:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 20 18:05:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 20 18:05:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> Apr 20 18:05:14 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 20 18:05:14 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 20 18:05:14 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 20 18:05:14 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 20 18:05:14 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Apr 20 18:05:14 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Apr 20 18:05:14 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Apr 20 18:05:14 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Apr 20 18:05:14 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Apr 20 18:05:14 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Apr 20 18:05:14 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Apr 20 18:05:14 at
>
[jira] [Commented] (FLINK-22457) KafkaSourceLegacyITCase.testMultipleSourcesOnePartition fails because of timeout
[
https://issues.apache.org/jira/browse/FLINK-22457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367812#comment-17367812
]
Xintong Song commented on FLINK-22457:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19331=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6593
> KafkaSourceLegacyITCase.testMultipleSourcesOnePartition fails because of
> timeout
>
>
> Key: FLINK-22457
> URL: https://issues.apache.org/jira/browse/FLINK-22457
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17140=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=80a658d1-f7f6-5d93-2758-53ac19fd5b19=7045
> {code:java}
> Apr 24 23:47:33 [ERROR] Tests run: 21, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 174.335 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase
> Apr 24 23:47:33 [ERROR]
> testMultipleSourcesOnePartition(org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase)
> Time elapsed: 60.019 s <<< ERROR!
> Apr 24 23:47:33 org.junit.runners.model.TestTimedOutException: test timed out
> after 6 milliseconds
> Apr 24 23:47:33 at sun.misc.Unsafe.park(Native Method)
> Apr 24 23:47:33 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Apr 24 23:47:33 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Apr 24 23:47:33 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> Apr 24 23:47:33 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Apr 24 23:47:33 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Apr 24 23:47:33 at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
> Apr 24 23:47:33 at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:1112)
> Apr 24 23:47:33 at
> org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testMultipleSourcesOnePartition(KafkaSourceLegacyITCase.java:87)
> Apr 24 23:47:33 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 24 23:47:33 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 24 23:47:33 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 24 23:47:33 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 24 23:47:33 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 24 23:47:33 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 24 23:47:33 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 24 23:47:33 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 24 23:47:33 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Apr 24 23:47:33 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Apr 24 23:47:33 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Apr 24 23:47:33 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15493) FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator failed on travis
[
https://issues.apache.org/jira/browse/FLINK-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367811#comment-17367811
]
Xintong Song commented on FLINK-15493:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19323=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6381
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis
> ---
>
> Key: FLINK-15493
> URL: https://issues.apache.org/jira/browse/FLINK-15493
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0, 1.13.0, 1.14.0
>Reporter: Dian Fu
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.14.0, 1.13.2
>
>
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis with the following exception:
> {code}
> Test
> testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> failed with: org.junit.runners.model.TestTimedOutException: test timed out
> after 3 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:177)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:115)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:197)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(FlinkKafkaInternalProducerITCase.java:176)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> instance: [https://api.travis-ci.org/v3/job/633307060/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15493) FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator failed on travis
[
https://issues.apache.org/jira/browse/FLINK-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-15493:
-
Fix Version/s: 1.14.0
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis
> ---
>
> Key: FLINK-15493
> URL: https://issues.apache.org/jira/browse/FLINK-15493
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0, 1.13.0, 1.14.0
>Reporter: Dian Fu
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.14.0, 1.13.2
>
>
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis with the following exception:
> {code}
> Test
> testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> failed with: org.junit.runners.model.TestTimedOutException: test timed out
> after 3 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:177)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:115)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:197)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(FlinkKafkaInternalProducerITCase.java:176)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> instance: [https://api.travis-ci.org/v3/job/633307060/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15493) FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator failed on travis
[
https://issues.apache.org/jira/browse/FLINK-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-15493:
-
Affects Version/s: 1.14.0
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis
> ---
>
> Key: FLINK-15493
> URL: https://issues.apache.org/jira/browse/FLINK-15493
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0, 1.13.0, 1.14.0
>Reporter: Dian Fu
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.13.2
>
>
> FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator
> failed on travis with the following exception:
> {code}
> Test
> testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> failed with: org.junit.runners.model.TestTimedOutException: test timed out
> after 3 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:177)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:115)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:197)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testProducerWhenCommitEmptyPartitionsToOutdatedTxnCoordinator(FlinkKafkaInternalProducerITCase.java:176)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> {code}
> instance: [https://api.travis-ci.org/v3/job/633307060/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (FLINK-22593) SavepointITCase.testShouldAddEntropyToSavepointPath unstable
[
https://issues.apache.org/jira/browse/FLINK-22593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reopened FLINK-22593:
--
Reopening.
Another instance on master, after the fix being merged.
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19303=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=4099
cc [~akalashnikov][~pnowojski]
> SavepointITCase.testShouldAddEntropyToSavepointPath unstable
>
>
> Key: FLINK-22593
> URL: https://issues.apache.org/jira/browse/FLINK-22593
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Robert Metzger
>Assignee: Anton Kalashnikov
>Priority: Blocker
> Labels: pull-request-available, stale-blocker, stale-critical,
> test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=9072=logs=cc649950-03e9-5fae-8326-2f1ad744b536=51cab6ca-669f-5dc0-221d-1e4f7dc4fc85
> {code}
> 2021-05-07T10:56:20.9429367Z May 07 10:56:20 [ERROR] Tests run: 13, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 33.441 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.SavepointITCase
> 2021-05-07T10:56:20.9445862Z May 07 10:56:20 [ERROR]
> testShouldAddEntropyToSavepointPath(org.apache.flink.test.checkpointing.SavepointITCase)
> Time elapsed: 2.083 s <<< ERROR!
> 2021-05-07T10:56:20.9447106Z May 07 10:56:20
> java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> triggering task Sink: Unnamed (3/4) of job 4e155a20f0a7895043661a6446caf1cb
> has not being executed at the moment. Aborting checkpoint. Failure reason:
> Not all required tasks are currently running.
> 2021-05-07T10:56:20.9448194Z May 07 10:56:20 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2021-05-07T10:56:20.9448797Z May 07 10:56:20 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2021-05-07T10:56:20.9449428Z May 07 10:56:20 at
> org.apache.flink.test.checkpointing.SavepointITCase.submitJobAndTakeSavepoint(SavepointITCase.java:305)
> 2021-05-07T10:56:20.9450160Z May 07 10:56:20 at
> org.apache.flink.test.checkpointing.SavepointITCase.testShouldAddEntropyToSavepointPath(SavepointITCase.java:273)
> 2021-05-07T10:56:20.9450785Z May 07 10:56:20 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-07T10:56:20.9451331Z May 07 10:56:20 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-07T10:56:20.9451940Z May 07 10:56:20 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-07T10:56:20.9452498Z May 07 10:56:20 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-07T10:56:20.9453247Z May 07 10:56:20 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-07T10:56:20.9454007Z May 07 10:56:20 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-07T10:56:20.9454687Z May 07 10:56:20 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-07T10:56:20.9455302Z May 07 10:56:20 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-07T10:56:20.9455909Z May 07 10:56:20 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-05-07T10:56:20.9456493Z May 07 10:56:20 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-07T10:56:20.9457074Z May 07 10:56:20 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-07T10:56:20.9457636Z May 07 10:56:20 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-07T10:56:20.9458157Z May 07 10:56:20 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-07T10:56:20.9458678Z May 07 10:56:20 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-07T10:56:20.9459252Z May 07 10:56:20 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-07T10:56:20.9459865Z May 07 10:56:20 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-07T10:56:20.9460433Z May 07 10:56:20 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-07T10:56:20.9461058Z May 07 10:56:20 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-07T10:56:20.9461607Z May 07 10:56:20 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[GitHub] [flink] xintongsong commented on pull request #16233: [FLINK-22085][tests] Add debug code to expose the sink status for KafkaSourceLegacyITCase
xintongsong commented on pull request #16233: URL: https://github.com/apache/flink/pull/16233#issuecomment-866469830 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16220: [FLINK-23033][python] Add the missing ObjectArrayTypeInfo
flinkbot edited a comment on pull request #16220: URL: https://github.com/apache/flink/pull/16220#issuecomment-865041649 ## CI report: * ee3f8975908e9f243bb5510d0cb09a8ddf341f7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19315) * cf469ce915e4f73fbd67d24c96d6190b113ec525 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #16240: [FLINK-23092][python] Fix built-in functions are unsupported in Python Group Window UDAF
HuangXingBo commented on a change in pull request #16240:
URL: https://github.com/apache/flink/pull/16240#discussion_r656705986
##
File path: flink-python/pyflink/table/tests/test_udaf.py
##
@@ -543,24 +543,25 @@ def test_tumbling_group_window_over_time(self):
from pyflink.testing import source_sink_utils
table_sink = source_sink_utils.TestAppendSink(
-['a', 'b', 'c', 'd'],
+['a', 'b', 'c', 'd', 'e'],
[
DataTypes.TINYINT(),
DataTypes.TIMESTAMP(3),
DataTypes.TIMESTAMP(3),
+DataTypes.BIGINT(),
DataTypes.BIGINT()])
self.t_env.register_table_sink("Results", table_sink)
t.window(Tumble.over("1.hours").on("rowtime").alias("w")) \
.group_by("a, w") \
-.select("a, w.start, w.end, my_count(c) as c") \
+.select("a, w.start, w.end, COUNT(c) as c, my_count(c) as d") \
.execute_insert("Results") \
.wait()
actual = source_sink_utils.results()
self.assert_equals(actual,
- ["+I[2, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 1]",
-"+I[3, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 1]",
-"+I[1, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 2]",
-"+I[1, 2018-03-11 04:00:00.0, 2018-03-11
05:00:00.0, 1]"])
+ ["+I[2, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 2, 1]",
Review comment:
In fact, the user-defined udaf is a count(distinct). so the result is
expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16146: [FLINK-22927][python] Fix the bug of JobStatus
flinkbot edited a comment on pull request #16146: URL: https://github.com/apache/flink/pull/16146#issuecomment-859379229 ## CI report: * 2c66d73b141988c8c2e2aa7b92fa2df763e2a06f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18970) * ce5eeaa31e866ed8950567b10ec973f85aef8d7a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #16240: [FLINK-23092][python] Fix built-in functions are unsupported in Python Group Window UDAF
HuangXingBo commented on a change in pull request #16240:
URL: https://github.com/apache/flink/pull/16240#discussion_r656705464
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -77,7 +77,11 @@ public boolean matches(RelOptRuleCall call) {
aggCalls.stream()
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
-aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+aggCalls.stream()
Review comment:
Currently we don't support general python udaf in stream over window and
batch mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #16240: [FLINK-23092][python] Fix built-in functions are unsupported in Python Group Window UDAF
HuangXingBo commented on a change in pull request #16240:
URL: https://github.com/apache/flink/pull/16240#discussion_r656705464
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -77,7 +77,11 @@ public boolean matches(RelOptRuleCall call) {
aggCalls.stream()
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
-aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+aggCalls.stream()
Review comment:
Currently we doesn't support general python udaf in stream over window
and batch mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15531: [FLINK-22147][connector/kafka] Refactor partition discovery logic in Kafka source enumerator
flinkbot edited a comment on pull request #15531: URL: https://github.com/apache/flink/pull/15531#issuecomment-816343442 ## CI report: * ef0489f55f950c18c3627296fd8db333a0e0d6af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18807) * 1503664338eb8cd7310e2c8a909c3599ffd72573 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #16240: [FLINK-23092][python] Fix built-in functions are unsupported in Python Group Window UDAF
dianfu commented on a change in pull request #16240:
URL: https://github.com/apache/flink/pull/16240#discussion_r656703620
##
File path: flink-python/pyflink/table/tests/test_udaf.py
##
@@ -543,24 +543,25 @@ def test_tumbling_group_window_over_time(self):
from pyflink.testing import source_sink_utils
table_sink = source_sink_utils.TestAppendSink(
-['a', 'b', 'c', 'd'],
+['a', 'b', 'c', 'd', 'e'],
[
DataTypes.TINYINT(),
DataTypes.TIMESTAMP(3),
DataTypes.TIMESTAMP(3),
+DataTypes.BIGINT(),
DataTypes.BIGINT()])
self.t_env.register_table_sink("Results", table_sink)
t.window(Tumble.over("1.hours").on("rowtime").alias("w")) \
.group_by("a, w") \
-.select("a, w.start, w.end, my_count(c) as c") \
+.select("a, w.start, w.end, COUNT(c) as c, my_count(c) as d") \
.execute_insert("Results") \
.wait()
actual = source_sink_utils.results()
self.assert_equals(actual,
- ["+I[2, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 1]",
-"+I[3, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 1]",
-"+I[1, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 2]",
-"+I[1, 2018-03-11 04:00:00.0, 2018-03-11
05:00:00.0, 1]"])
+ ["+I[2, 2018-03-11 03:00:00.0, 2018-03-11
04:00:00.0, 2, 1]",
Review comment:
It is a little wired that the results are not equal. Is this expected?
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -77,7 +77,11 @@ public boolean matches(RelOptRuleCall call) {
aggCalls.stream()
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
-aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+aggCalls.stream()
Review comment:
Does the issue also exist in StreamPhysicalPythonOverAggregateRule,
BatchPhysicalPythonAggregateRule, BatchPhysicalPythonWindowAggregateRule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16245: [FLINK-22940][SQL-CLIENT] Make sql client column max width configurable
flinkbot edited a comment on pull request #16245: URL: https://github.com/apache/flink/pull/16245#issuecomment-866278518 ## CI report: * 7e0f7cb3f507be092d334a8133bcb3daac4b9f03 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19334) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14839: [FLINK-21353][state] Add DFS-based StateChangelog
flinkbot edited a comment on pull request #14839: URL: https://github.com/apache/flink/pull/14839#issuecomment-772060196 ## CI report: * 426533428e0971d34f6e80acc89fe5a5a72ea2a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15638) * d6adb94d3584e2d1a66e54178a006b36aea1970c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19341) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14839: [FLINK-21353][state] Add DFS-based StateChangelog
flinkbot edited a comment on pull request #14839: URL: https://github.com/apache/flink/pull/14839#issuecomment-772060196 ## CI report: * 426533428e0971d34f6e80acc89fe5a5a72ea2a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15638) * d6adb94d3584e2d1a66e54178a006b36aea1970c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-8438) Ensure that implicit Scala TypeInformation works in Eclipse IDE
[ https://issues.apache.org/jira/browse/FLINK-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8438: -- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Ensure that implicit Scala TypeInformation works in Eclipse IDE > --- > > Key: FLINK-8438 > URL: https://issues.apache.org/jira/browse/FLINK-8438 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Reporter: Timo Walther >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > It seems that Scala macros do not work properly in the Eclipse IDE. We need > to investigate this further and maybe need to update the Scala macros. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8566) Replace retract/insert of same record for state retention timer resets
[
https://issues.apache.org/jira/browse/FLINK-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8566:
--
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Replace retract/insert of same record for state retention timer resets
> --
>
> Key: FLINK-8566
> URL: https://issues.apache.org/jira/browse/FLINK-8566
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.5.0
>Reporter: Fabian Hueske
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> Currently a simple query like {{SELECT DISTINCT a, b, c FROM tableX}} is
> translated into a plan that generates a retraction stream. However, one would
> assume that an append stream should be possible as well. In fact, the plan
> doesn't produce actual updates.
> Internally, the {{DISTINCT}} is translated into a {{GROUP BY}} with all
> distinct fields being keys and no aggregation functions. The corresponding
> operator produces updates, because aggregation function might update their
> results as new records are received. So we could just implement a dedicated
> operator for {{DISTINCT}}. However, this would not work if a user configures
> a state retention time. In this case, we emit retraction/insert messages for
> the same (distinct) record whenever a new row is received to reset the state
> clean-up timers of the downstream operators.
> One way to solve this issue to implement a dedicated mechanism to update
> state clean-up timers for unchanged records instead of sending out
> retraction/insert messages with identical records. This mechanism would just
> be used to reset the timers and could also be used for append streams. For
> example, we could replace the boolean flag in CRow with a byte that can take
> more than two values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22443) can not be execute an extreme long sql under batch mode
[
https://issues.apache.org/jira/browse/FLINK-22443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22443:
---
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as a
Blocker but is unassigned and neither itself nor its Sub-Tasks have been
updated for 1 days. I have gone ahead and marked it "stale-blocker". If this
ticket is a Blocker, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> can not be execute an extreme long sql under batch mode
> ---
>
> Key: FLINK-22443
> URL: https://issues.apache.org/jira/browse/FLINK-22443
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.2
> Environment: execute command
>
> {code:java}
> bin/sql-client.sh embedded -d conf/sql-client-batch.yaml
> {code}
> content of conf/sql-client-batch.yaml
>
> {code:java}
> catalogs:
> - name: bnpmphive
> type: hive
> hive-conf-dir: /home/gum/hive/conf
> hive-version: 3.1.2
> execution:
> planner: blink
> type: batch
> #type: streaming
> result-mode: table
> parallelism: 4
> max-parallelism: 2000
> current-catalog: bnpmphive
> #current-database: snmpprobe
> #configuration:
> # table.sql-dialect: hivemodules:
>- name: core
> type: core
>- name: myhive
> type: hivedeployment:
> # general cluster communication timeout in ms
> response-timeout: 5000
> # (optional) address from cluster to gateway
> gateway-address: ""
> # (optional) port from cluster to gateway
> gateway-port: 0
> {code}
>
>Reporter: macdoor615
>Priority: Blocker
> Labels: stale-blocker, stale-critical
> Attachments: flink-gum-taskexecutor-8-hb3-prod-hadoop-002.log.4.zip,
> raw_p_restapi_hcd.csv.zip
>
>
> 1. execute an extreme long sql under batch mode
>
> {code:java}
> select
> 'CD' product_name,
> r.code business_platform,
> 5 statisticperiod,
> cast('2021-03-24 00:00:00' as timestamp) coltime,
> cast(r1.indicatorvalue as double) as YWPT_ZHQI_CD_038_GZ_2,
> cast(r2.indicatorvalue as double) as YWPT_ZHQI_CD_038_YW_7,
> cast(r3.indicatorvalue as double) as YWPT_ZHQI_CD_038_YW_5,
> cast(r4.indicatorvalue as double) as YWPT_ZHQI_CD_038_YW_6,
> cast(r5.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00029,
> cast(r6.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00028,
> cast(r7.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00015,
> cast(r8.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00014,
> cast(r9.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00011,
> cast(r10.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00010,
> cast(r11.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00013,
> cast(r12.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00012,
> cast(r13.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00027,
> cast(r14.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00026,
> cast(r15.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00046,
> cast(r16.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00047,
> cast(r17.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00049,
> cast(r18.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00048,
> cast(r19.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00024,
> cast(r20.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00025,
> cast(r21.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00022,
> cast(r22.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00023,
> cast(r23.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00054,
> cast(r24.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00055,
> cast(r25.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00033,
> cast(r26.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00032,
> cast(r27.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00053,
> cast(r28.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00052,
> cast(r29.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00051,
> cast(r30.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00050,
> cast(r31.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00043,
> cast(r32.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00042,
> cast(r33.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00017,
> cast(r34.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00016,
> cast(r35.indicatorvalue as double) as YWPT_ZHQI_CD_038_GZ_3,
> cast(r36.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00045,
> cast(r37.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00044,
> cast(r38.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00038,
> cast(r39.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00039,
> cast(r40.indicatorvalue as double) as YWPT_ZHQI_CD_038_XT_00037,
>
[jira] [Updated] (FLINK-22996) The description about coalesce is wrong
[
https://issues.apache.org/jira/browse/FLINK-22996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22996:
---
Labels: stale-critical (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> The description about coalesce is wrong
> ---
>
> Key: FLINK-22996
> URL: https://issues.apache.org/jira/browse/FLINK-22996
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.13.0
>Reporter: lothar
>Priority: Critical
> Labels: stale-critical
> Attachments: image-2021-06-15-17-14-56-270.png
>
>
> In Flink website, there is a fault that the description about coalesce
> function.The document link is
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/functions/systemfunctions|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/functions/systemfunctions.]
> !image-2021-06-15-17-14-56-270.png!
>
> It should be “{{COALESCE(NULL, 5)}} returns 5”.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22982) java.lang.ClassCastException when using Python UDF
[
https://issues.apache.org/jira/browse/FLINK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22982:
---
Labels: pull-request-available stale-critical (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> java.lang.ClassCastException when using Python UDF
> --
>
> Key: FLINK-22982
> URL: https://issues.apache.org/jira/browse/FLINK-22982
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.1
>Reporter: Maciej Bryński
>Priority: Critical
> Labels: pull-request-available, stale-critical
> Fix For: 1.14.0, 1.13.2
>
>
> Hi,
> I'm trying to use Python UDF with logical condition as argument.
>
> {code:java}
> log = logging.getLogger()
> @udf(result_type=DataTypes.BOOLEAN())
> def trace(message, condition):
> if condition:
> log.warn(message)
> return condition
> table_env.create_temporary_function('trace', trace)
> table_env.execute_sql("""
> CREATE TABLE datagen (
> n int
> ) WITH (
> 'connector' = 'datagen',
> 'number-of-rows' = '10'
> )
> """)
> result = table_env.sql_query("""
> SELECT *
> FROM datagen
> WHERE trace(n, n < 0)
> """)
> for r in result.execute().collect():
> print(r){code}
>
> As a result I'm getting exception:
> {code:java}
> Py4JJavaError: An error occurred while calling o135.execute.
> : java.lang.ClassCastException: class org.apache.calcite.rex.RexInputRef
> cannot be cast to class org.apache.calcite.rex.RexCall
> (org.apache.calcite.rex.RexInputRef and org.apache.calcite.rex.RexCall are in
> unnamed module of loader 'app')
> at
> org.apache.flink.table.planner.plan.rules.logical.PythonMapMergeRule.matches(PythonMapMergeRule.java:70)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:538)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22898: --- Labels: auto-deprioritized-critical (was: stale-critical) Priority: Major (was: Critical) This issue was labeled "stale-critical" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Critical, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Priority: Major > Labels: auto-deprioritized-critical > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22386) Introduce cache for docker images
[ https://issues.apache.org/jira/browse/FLINK-22386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22386: --- Labels: stale-critical (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Critical but is unassigned and neither itself nor its Sub-Tasks have been updated for 7 days. I have gone ahead and marked it "stale-critical". If this ticket is critical, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Introduce cache for docker images > - > > Key: FLINK-22386 > URL: https://issues.apache.org/jira/browse/FLINK-22386 > Project: Flink > Issue Type: Improvement > Components: Build System / Azure Pipelines >Reporter: Dawid Wysakowicz >Priority: Critical > Labels: stale-critical > > Tests that use testcontainers occassionally fail when downloading images from > docker hub. It would be nice to have a caching local proxy to the docker hub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-4860) Sort performance
[ https://issues.apache.org/jira/browse/FLINK-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-4860: -- Labels: performance stale-major (was: performance) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Sort performance > > > Key: FLINK-4860 > URL: https://issues.apache.org/jira/browse/FLINK-4860 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: Greg Hogan >Priority: Major > Labels: performance, stale-major > > A super-task for improvements to Flink's sort performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22897) FlinkSQL1.12 Sink to Hive with diffrent parallelism will due to produce many small files
[
https://issues.apache.org/jira/browse/FLINK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22897:
---
Labels: auto-deprioritized-critical (was: stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> FlinkSQL1.12 Sink to Hive with diffrent parallelism will due to produce many
> small files
>
>
> Key: FLINK-22897
> URL: https://issues.apache.org/jira/browse/FLINK-22897
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.12.1
>Reporter: zhengjiewen
>Priority: Major
> Labels: auto-deprioritized-critical
> Fix For: 1.14.0
>
>
> I try to use flink sql in batch mode, to sink data into hive partition table,
> here is the sql:
>
> {code:java}
> //代码占位符
> INSERT OVERWRTITE 【targetTable】SELECT 【field】FROM 【sourceTable】;
> {code}
>
>
>
> And I found that when the parallelism of the sink operator is different from
> that of the operator before it, a large number of small files will be
> generated. But this is not the case when the parallelism is the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-3989) Add guide to write a custom streaming connector in DataStream API documentation
[ https://issues.apache.org/jira/browse/FLINK-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-3989: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add guide to write a custom streaming connector in DataStream API > documentation > --- > > Key: FLINK-3989 > URL: https://issues.apache.org/jira/browse/FLINK-3989 > Project: Flink > Issue Type: New Feature > Components: Documentation >Affects Versions: 1.1.0 >Reporter: Tzu-Li (Gordon) Tai >Priority: Major > Labels: auto-unassigned, stale-major > > I think it's a good idea to add a basic outline in the documentation for > users to compose their own streaming connector source / sinks, as its quite a > frequently asked question in the mailing lists and might be helpful for Flink > to get traction on new external systems faster. > The documentation should contain information about what classes to extend, > what life cycle methods to implement, how to implement an exactly once > source, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8625) Move OutputFlusher thread to Netty scheduled executor
[ https://issues.apache.org/jira/browse/FLINK-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8625: -- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Move OutputFlusher thread to Netty scheduled executor > - > > Key: FLINK-8625 > URL: https://issues.apache.org/jira/browse/FLINK-8625 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Piotr Nowojski >Priority: Major > Labels: auto-unassigned, stale-major > > This will allow us to trigger/schedule next flush only if we are not > currently busy. > PR: https://github.com/apache/flink/pull/6698 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8474) Add documentation for HBaseTableSource
[
https://issues.apache.org/jira/browse/FLINK-8474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8474:
--
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Add documentation for HBaseTableSource
> --
>
> Key: FLINK-8474
> URL: https://issues.apache.org/jira/browse/FLINK-8474
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Table SQL / Ecosystem
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Fabian Hueske
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> The {{HBaseTableSource}} is not documented in the [Table Source and Sinks
> documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/sourceSinks.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11323) Upgrade KryoSerializer snapshot to implement new TypeSerializerSnapshot interface
[ https://issues.apache.org/jira/browse/FLINK-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-11323: --- Labels: auto-unassigned pull-request-available stale-major (was: auto-unassigned pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Upgrade KryoSerializer snapshot to implement new TypeSerializerSnapshot > interface > - > > Key: FLINK-11323 > URL: https://issues.apache.org/jira/browse/FLINK-11323 > Project: Flink > Issue Type: Technical Debt > Components: API / Type Serialization System >Reporter: Igal Shilman >Priority: Major > Labels: auto-unassigned, pull-request-available, stale-major > Time Spent: 20m > Remaining Estimate: 0h > > This issue is part of the ongoing effort to migrate the existing serializers > and their associated snapshot classes to the new snapshot abstraction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-8293) Rework Flink's type and serialization docs
[
https://issues.apache.org/jira/browse/FLINK-8293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8293:
--
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Rework Flink's type and serialization docs
> --
>
> Key: FLINK-8293
> URL: https://issues.apache.org/jira/browse/FLINK-8293
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System, Documentation
>Reporter: Timo Walther
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> The documentation about Flink's type and serialization system hasn't been
> updated for a while and there are a lot of users (especially beginners) that
> have problems with defining types for inputs, functions, state etc. We should
> rework the documentation a little bit to solve things like:
> Type related things:
> {code}
> "Document all data types.
> What TypeInfo is available? What are the limitation? Encourage TypeHints?
> returns()? Link to new Types class. How to declare a valid case class in
> Scala.
> Look into log if type is no POJO type (e.g. when using org.json4s)
> ResultTypeQueryable documentation
> Case classes and Tuples do not support null!
> Subtypes of POJOs are handled during runtime or via cache and registerType()
> Explain all methods in ExecutionConfig.
> Compatibility guarantees.
> Pojos must have a void setter. Why are we so strict?
> Update docs in api_concepts about types (Avro is not used for POJOs)!"
> {code}
> Serialization related things:
> {code}
> "Serialization overview. Big picture (what is serialized, how, why, where,
> when?).
> When/why should I register a type or a subtype -- what does that do?
> Link to ""Streaming/Working with State/Custom Serialization for Managed
> State""."
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11222) Change api.scala.DataStream to api.datastream.DataStream for createHarnessTester in HarnessTestBase
[
https://issues.apache.org/jira/browse/FLINK-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11222:
---
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Change api.scala.DataStream to api.datastream.DataStream for
> createHarnessTester in HarnessTestBase
> ---
>
> Key: FLINK-11222
> URL: https://issues.apache.org/jira/browse/FLINK-11222
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Legacy Planner
>Reporter: Hequn Cheng
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Thanks to FLINK-11074, we can create harness tester from a DataStream which
> makes easier to write harness test.
> However, it would be better if we change the parameter type from
> api.scala.DataStream to api.datastream.DataStream for the
> \{{createHarnessTester()}} method, so that both java.DataStream and
> scala.DataStream can use this method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11007) Update documentation to describe new checkpoint metadata file behavior
[ https://issues.apache.org/jira/browse/FLINK-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-11007: --- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Update documentation to describe new checkpoint metadata file behavior > -- > > Key: FLINK-11007 > URL: https://issues.apache.org/jira/browse/FLINK-11007 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / Checkpointing >Reporter: Josh Lemer >Priority: Major > Labels: auto-unassigned, stale-major > > In the > [documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/checkpointing.html#state-checkpoints-dir] > about checkpointing, it is explained that you must set the config file > setting `state.checkpoints.dir` to specify the directory where checkpoint > metadata files will be stored. > This is no longer the case, and apparently now checkpoint metadata files are > stored in the checkpoint data directory itself. This should be updated in the > docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
