[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18033: [FLINK-25141][connector/elasticsearch] add sink parallelism option

[GitBox]

flinkbot edited a comment on pull request #18033:
URL: https://github.com/apache/flink/pull/18033#issuecomment-987520681


   
   ## CI report:
   
   * e74cadfcf7b3f6a66548c81c11d95c61dacd5c07 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27684)
 
   * 5b443ef1754daaffbde039149ab2aef0ec228cff UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-25398) Show complete stacktrace when requesting thread dump

[Xintong Song (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463053#comment-17463053
 ] 

Xintong Song commented on FLINK-25398:
--

Hi [~zuston],

Thanks for bringing this up. However, I'm not sure removing the stack depth 
limit is always an improvement for this feature. My gut feeling this will be 
improving the experience in some cases where deeper stack information is 
needed, at the cost of making the experience worse in many other use cases.

While providing complete information, removing the limit may also slow down 
generating the thread dump, bug users with redundant information and slow down 
the web ui. I think that's probably why JDK decides not to print the complete 
thread stack in the first place, and I'm in favor of respecting the judgement 
of the JDK team.

> Show complete stacktrace when requesting thread dump
> 
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
>  Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
>  Labels: pull-request-available
> Attachments: stacktrace.png
>
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's 
> threaddump  in runtime webui. Hence it's hard to the initial calling 
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk: 
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>  
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using 
> {{ThreadInfo.toString}} directly
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-25239) Delete useless variables

[Ada Wong (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ada Wong updated FLINK-25239:
-
Component/s: Table SQL / Ecosystem

> Delete useless variables
> 
>
> Key: FLINK-25239
> URL: https://issues.apache.org/jira/browse/FLINK-25239
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / JDBC, Table SQL / Ecosystem
>Affects Versions: 1.14.0
>Reporter: Ada Wong
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.15.0, 1.14.3
>
>
>    DEFAULT_FLUSH_MAX_SIZE and DEFAULT_FLUSH_INTERVAL_MILLS in 
> JdbcOutputFormat class are useless.
> We should delete them.
>  
> {code:java}
> public static final int DEFAULT_FLUSH_MAX_SIZE = 5000;
> public static final long DEFAULT_FLUSH_INTERVAL_MILLS = 0L; {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] godfreyhe commented on a change in pull request #17845: [FLINK-24352] [flink-table-planner] Add null check for temporal table check on SqlSnapshot

[GitBox]

godfreyhe commented on a change in pull request #17845:
URL: https://github.com/apache/flink/pull/17845#discussion_r772888377



##
File path: 
flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
##
@@ -4711,6 +4711,11 @@ private void validateSnapshot(SqlNode node, 
SqlValidatorScope scope, SqlValidato
 Static.RESOURCE.illegalExpressionForTemporal(
 dataType.getSqlTypeName().getName()));
 }
+if (ns instanceof IdentifierNamespace && ns.resolve() instanceof 
WithItemNamespace) {
+// If the snapshot is used over a CTE, then we don't have a 
concrete underlying
+// table to operate on. This will be rechecked later in the 
planner rules.
+return;

Review comment:
   @YuvalItzchakov Y I think you can throw `CalciteContextException` 
directly, just call `newValidationError()`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

flinkbot edited a comment on pull request #18157:
URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589


   
   ## CI report:
   
   * 0f1c86c86ba5a4c3388ef2aac95af103a3bfcced Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28409)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread

[GitBox]

flinkbot edited a comment on pull request #17988:
URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654


   
   ## CI report:
   
   * e7be85162e0b431d518ab1ffe8b59283338d00b7 Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28411)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   * 74c203d84576d6a8c9406a42ffb82cb9cfba5e81 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] ZhangChaoming commented on a change in pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

ZhangChaoming commented on a change in pull request #18151:
URL: https://github.com/apache/flink/pull/18151#discussion_r772879377



##
File path: 
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/table/DataGeneratorConnectorITCase.java
##
@@ -69,6 +73,33 @@ public void testTypes() throws Exception {
 }
 }
 
-Assert.assertEquals("Unexpected number of results", 10, 
results.size());
+assertThat(results.size()).isEqualTo(10);
+}
+
+@Test
+public void testLimit() throws Exception {
+final TestingTableEnvironment env = TestingTableEnvironment
+.create(
+
EnvironmentSettings.newInstance().inStreamingMode().build(),
+null,
+new TableConfig());
+
+env.executeSql("CREATE TABLE datagen_t (\n"
++ "f0 CHAR(1)\n"
++ ") WITH ("
++ "'connector' = 'datagen'"
++ ")");
+
+List results = new ArrayList<>();
+
+try (CloseableIterator iter = env
+.executeSql("select * from datagen_t limit 5")
+.collect()) {
+while (iter.hasNext()) {
+results.add(iter.next());
+}
+}
+
+assertThat(results.size()).isEqualTo(5);

Review comment:
   OK




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] ZhangChaoming commented on a change in pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

ZhangChaoming commented on a change in pull request #18151:
URL: https://github.com/apache/flink/pull/18151#discussion_r772878789



##
File path: 
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/table/DataGeneratorConnectorITCase.java
##
@@ -69,6 +73,33 @@ public void testTypes() throws Exception {
 }
 }
 
-Assert.assertEquals("Unexpected number of results", 10, 
results.size());
+assertThat(results.size()).isEqualTo(10);
+}
+
+@Test
+public void testLimit() throws Exception {
+final TestingTableEnvironment env = TestingTableEnvironment
+.create(
+
EnvironmentSettings.newInstance().inStreamingMode().build(),
+null,
+new TableConfig());
+
+env.executeSql("CREATE TABLE datagen_t (\n"
++ "f0 CHAR(1)\n"
++ ") WITH ("
++ "'connector' = 'datagen'"
++ ")");
+
+List results = new ArrayList<>();
+
+try (CloseableIterator iter = env
+.executeSql("select * from datagen_t limit 5")

Review comment:
   Maybe we can inffer that from it's logical plan.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28415)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector

[GitBox]

flinkbot edited a comment on pull request #18151:
URL: https://github.com/apache/flink/pull/18151#issuecomment-997659194


   
   ## CI report:
   
   * 4bd3f996754648fe8913502f17698636619e8b33 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28381)
 
   * 15cb79a2b8be65c67adf2825fbca797df85da1ef UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18112: [FLINK-25305][checkpoint] Always wait for input channel state and result partition state get completed in AsyncRunnable

[GitBox]

flinkbot edited a comment on pull request #18112:
URL: https://github.com/apache/flink/pull/18112#issuecomment-994403719


   
   ## CI report:
   
   * 8cbb17363a0bff7485beecad8d2d375bfe67b0e2 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28201)
 
   * be0e1ad15b08a9b5932ea282e215cd53ca557f5e Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28414)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18112: [FLINK-25305][checkpoint] Always wait for input channel state and result partition state get completed in AsyncRunnable

[GitBox]

flinkbot edited a comment on pull request #18112:
URL: https://github.com/apache/flink/pull/18112#issuecomment-994403719


   
   ## CI report:
   
   * 8cbb17363a0bff7485beecad8d2d375bfe67b0e2 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28201)
 
   * be0e1ad15b08a9b5932ea282e215cd53ca557f5e UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] gaoyunhaii commented on pull request #18112: [FLINK-25305][checkpoint] Always wait for input channel state and result partition state get completed in AsyncRunnable

[GitBox]

gaoyunhaii commented on pull request #18112:
URL: https://github.com/apache/flink/pull/18112#issuecomment-998522990


   Very thanks @pnowojski for the review! I updated the PR and the commit 
message~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread

[GitBox]

flinkbot edited a comment on pull request #17988:
URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654


   
   ## CI report:
   
   * f015c1d120d1b63a3014126ff28b07cff58fc9c0 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28039)
 
   * e7be85162e0b431d518ab1ffe8b59283338d00b7 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28411)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread

[GitBox]

wuchong commented on pull request #17988:
URL: https://github.com/apache/flink/pull/17988#issuecomment-998522664


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17734: [FLINK-23305][table]Support semi/anti interval joins

[GitBox]

flinkbot edited a comment on pull request #17734:
URL: https://github.com/apache/flink/pull/17734#issuecomment-964006875


   
   ## CI report:
   
   * 6e2c662fc3a4e8d63bbf1ffe2b910571b11fc3fe Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28405)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-ml] lindong28 commented on a change in pull request #41: [hotfix][FLIP-176] Updates onEpochWatermarkIncremented() and onIterationTerminated() to throw Exception

[GitBox]

lindong28 commented on a change in pull request #41:
URL: https://github.com/apache/flink-ml/pull/41#discussion_r772871230



##
File path: 
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/logisticregression/LogisticRegression.java
##
@@ -362,7 +362,6 @@ private void updateModel() {
 @Override
 public void onEpochWatermarkIncremented(
 int epochWatermark, Context context, Collector 
collector) {
-// TODO: let this method throws exception.

Review comment:
   Thanks for pointing out this issue. The issue has been fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-ml] lindong28 commented on a change in pull request #41: [hotfix][FLIP-176] Updates onEpochWatermarkIncremented() and onIterationTerminated() to throw Exception

[GitBox]

lindong28 commented on a change in pull request #41:
URL: https://github.com/apache/flink-ml/pull/41#discussion_r772870368



##
File path: 
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/logisticregression/LogisticRegression.java
##
@@ -362,7 +362,6 @@ private void updateModel() {
 @Override
 public void onEpochWatermarkIncremented(
 int epochWatermark, Context context, Collector 
collector) {
-// TODO: let this method throws exception.

Review comment:
   hmm... why do we need to do that?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-16478) add restApi to modify loglevel

[Wenhao Ji (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463024#comment-17463024
 ] 

Wenhao Ji commented on FLINK-16478:
---

Thanks, [~trohrmann] !

> add restApi to modify loglevel 
> ---
>
> Key: FLINK-16478
> URL: https://issues.apache.org/jira/browse/FLINK-16478
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / REST
>Reporter: xiaodao
>Assignee: Wenhao Ji
>Priority: Minor
>  Labels: auto-deprioritized-major
>
> sometimes we may need to change loglevel to get more information to resolved 
> bug, now we need to stop it and modify conf/log4j.properties and resubmit it 
> ,i think it's better to add rest api to modify loglevel.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink-ml] lindong28 commented on a change in pull request #41: [hotfix][FLIP-176] Updates onEpochWatermarkIncremented() and onIterationTerminated() to throw Exception

[GitBox]

lindong28 commented on a change in pull request #41:
URL: https://github.com/apache/flink-ml/pull/41#discussion_r772870609



##
File path: 
flink-ml-iteration/src/main/java/org/apache/flink/iteration/Iterations.java
##
@@ -117,19 +117,16 @@
  */
 public static DataStreamList iterateUnboundedStreams(
 DataStreamList initVariableStreams, DataStreamList dataStreams, 
IterationBody body) {
-return createIteration(
+return iterateStreams(

Review comment:
   This change is reverted.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-ml] lindong28 commented on a change in pull request #41: [hotfix][FLIP-176] Updates onEpochWatermarkIncremented() and onIterationTerminated() to throw Exception

[GitBox]

lindong28 commented on a change in pull request #41:
URL: https://github.com/apache/flink-ml/pull/41#discussion_r772870536



##
File path: 
flink-ml-iteration/src/main/java/org/apache/flink/iteration/Iterations.java
##
@@ -261,10 +256,6 @@ private static DataStreamList createIteration(
 }
 
 List> tailsAndCriteriaTails = new 
ArrayList<>(tails.getDataStreams());
-checkState(

Review comment:
   This change is reverted.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink-ml] lindong28 commented on pull request #41: [hotfix][FLIP-176] Updates onEpochWatermarkIncremented() and onIterationTerminated() to throw Exception

[GitBox]

lindong28 commented on pull request #41:
URL: https://github.com/apache/flink-ml/pull/41#issuecomment-998518178


   @gaoyunhaii Could you help review this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] YuvalItzchakov commented on pull request #17845: [FLINK-24352] [flink-table-planner] Add null check for temporal table check on SqlSnapshot

[GitBox]

YuvalItzchakov commented on pull request #17845:
URL: https://github.com/apache/flink/pull/17845#issuecomment-998515048


   @tsreaper @godfreyhe @MartijnVisser Guys this has been waiting for a month 
now although the PR is ready. 
   
   This fix is affecting the ability of my consumers to use Flink and I'd 
really like to get this merged. 
   
   As there is no more critical issues re table nullability, is there anything 
else stalling us?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] hackergin commented on a change in pull request #18058: [FLINK-24571][connectors/elasticsearch] Support generating dynamic index based on proctime

[GitBox]

hackergin commented on a change in pull request #18058:
URL: https://github.com/apache/flink/pull/18058#discussion_r772863211



##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -240,6 +273,11 @@ boolean checkIsDynamicIndexWithFormat(String index) {
 return dynamicIndexTimeExtractPattern.matcher(index).matches();
 }
 
+/** Check generate a dynamic index is based on processing time. */
+boolean checkIsDynamicIndexWithProcTimeFormat(String index) {

Review comment:
   @leonardBang  I agree with you , and I'll change the describetion  of 
this issue/PR later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lipusheng commented on pull request #18147: [FLINK-25370][docs] update HBase SQL Connector Options table-name des…

[GitBox]

lipusheng commented on pull request #18147:
URL: https://github.com/apache/flink/pull/18147#issuecomment-998504439


   @AHeise Could you help merge it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-25398) Show complete stacktrace when requesting thread dump

[Junfan Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463009#comment-17463009
 ] 

Junfan Zhang commented on FLINK-25398:
--

Could you help check this ticket? [~guoyangze]  [~yunta]  [~xtsong] 

Besides optimized stacktrace has been attached in above attachment.

 

> Show complete stacktrace when requesting thread dump
> 
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
>  Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
>  Labels: pull-request-available
> Attachments: stacktrace.png
>
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's 
> threaddump  in runtime webui. Hence it's hard to the initial calling 
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk: 
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>  
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using 
> {{ThreadInfo.toString}} directly
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (FLINK-25398) Show complete stacktrace when requesting thread dump

[Junfan Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junfan Zhang updated FLINK-25398:
-
Attachment: stacktrace.png

> Show complete stacktrace when requesting thread dump
> 
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
>  Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
>  Labels: pull-request-available
> Attachments: stacktrace.png
>
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's 
> threaddump  in runtime webui. Hence it's hard to the initial calling 
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk: 
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>  
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using 
> {{ThreadInfo.toString}} directly
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot commented on pull request #18158: [FLINK-25398] Show complete stacktrace when requesting thread dump

[GitBox]

flinkbot commented on pull request #18158:
URL: https://github.com/apache/flink/pull/18158#issuecomment-998497916


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 8d454d78b2f90d8fe2cc0c78b4f4dd5eb7750ef9 (Tue Dec 21 
06:03:24 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-25398).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18158: [FLINK-25398] Show complete stacktrace when requesting thread dump

[GitBox]

flinkbot edited a comment on pull request #18158:
URL: https://github.com/apache/flink/pull/18158#issuecomment-998496692


   
   ## CI report:
   
   * 8d454d78b2f90d8fe2cc0c78b4f4dd5eb7750ef9 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28413)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #18158: [FLINK-25398] Show complete stacktrace when requesting thread dump

[GitBox]

flinkbot commented on pull request #18158:
URL: https://github.com/apache/flink/pull/18158#issuecomment-998496692


   
   ## CI report:
   
   * 8d454d78b2f90d8fe2cc0c78b4f4dd5eb7750ef9 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-25398) Show complete stacktrace when requesting thread dump

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-25398:
---
Labels: pull-request-available  (was: )

> Show complete stacktrace when requesting thread dump
> 
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
>  Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's 
> threaddump  in runtime webui. Hence it's hard to the initial calling 
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk: 
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>  
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using 
> {{ThreadInfo.toString}} directly
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] zuston opened a new pull request #18158: [FLINK-25398] Show complete stacktrace when requesting thread dump

[GitBox]

zuston opened a new pull request #18158:
URL: https://github.com/apache/flink/pull/18158


   
   ## What is the purpose of the change
   
   
   Now the stacktrace is not complete when clicking the task executor's 
threaddump  in runtime webui. Hence it's hard to the initial calling according 
to the stacktrace.
   
   Now the thread stacktrace is limited to 8, refer to openjdk: 
   
   
https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597
   
   ### Solution
   Using the custom stringify method to return stacktrace instead of using 
ThreadInfo.toString directly
   
   ## Verifying this change
   
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't 
know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-25398) Show complete stacktrace when requesting thread dump

[Junfan Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junfan Zhang updated FLINK-25398:
-
Description: 
h2. Why

Now the stacktrace is not complete when clicking the task executor's threaddump 
 in runtime webui. Hence it's hard to the initial calling according to the 
stacktrace.

Now the thread stacktrace is limited to 8, refer to openjdk: 

[https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]

 

h2. Solution
Using the custom {{stringify}} method to return stacktrace instead of using 
{{ThreadInfo.toString}} directly

 

> Show complete stacktrace when requesting thread dump
> 
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
>  Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's 
> threaddump  in runtime webui. Hence it's hard to the initial calling 
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk: 
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>  
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using 
> {{ThreadInfo.toString}} directly
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25398) Show complete stacktrace when requesting thread dump

[Junfan Zhang (Jira)]

Junfan Zhang created FLINK-25398:


 Summary: Show complete stacktrace when requesting thread dump
 Key: FLINK-25398
 URL: https://issues.apache.org/jira/browse/FLINK-25398
 Project: Flink
  Issue Type: Improvement
Reporter: Junfan Zhang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Closed] (FLINK-25378) kubernetes's request/limit resource can been seperated?

[Yang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yang Wang closed FLINK-25378.
-
Resolution: Duplicate

> kubernetes's request/limit resource can been seperated?
> ---
>
> Key: FLINK-25378
> URL: https://issues.apache.org/jira/browse/FLINK-25378
> Project: Flink
>  Issue Type: New Feature
>  Components: Deployment / Kubernetes
>Affects Versions: 1.13.2
> Environment: flink 1.13.2
> kubernetes
>Reporter: jeff-zou
>Priority: Major
>
> now my kubernetes cluster has very low CPU utilization,but cannt publish more 
> tasks,because resources are already occupied by Pod request.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Closed] (FLINK-25321) standalone deploy on k8s,pod always OOM killed,actual heap memory usage is normal, gc is normal

[Yang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yang Wang closed FLINK-25321.
-
Resolution: Information Provided

> standalone deploy on k8s,pod always OOM killed,actual heap memory usage is 
> normal, gc is normal
> ---
>
> Key: FLINK-25321
> URL: https://issues.apache.org/jira/browse/FLINK-25321
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.11.3
> Environment: Flink 1.11.3
> k8s v1.21.0
> standlone deployment
>Reporter: Gao Fei
>Priority: Major
>
> Start a cluster on k8s, deploy in standalone mode, a jobmanager pod (1G) and 
> a taskmanager pod (3372MB limit), the total memory configuration of the Flink 
> TM process is 3072MB, and the managed configuration is 0, both of which are 
> on the heap memory. Now the pod It will always be OOM killed, and the total 
> process memory will always exceed 3072MB. I saw that the system has adopted 
> jemlloc. There is no 64M problem. The application itself has not applied for 
> direct memory. It is strange why the process is always killed by OOM after a 
> period of time.
>  
> INFO  [] - Final TaskExecutor Memory configuration:
> INFO  [] -   Total Process Memory:          3.000gb (3221225472 bytes)
> INFO  [] -     Total Flink Memory:          2.450gb (2630667464 bytes)
> INFO  [] -       Total JVM Heap Memory:     2.080gb (2233382986 bytes)
> INFO  [] -         Framework:               128.000mb (134217728 bytes)
> INFO  [] -         Task:                    1.955gb (2099165258 bytes)
> INFO  [] -       Total Off-heap Memory:     378.880mb (397284478 bytes)
> INFO  [] -         Managed:                 0 bytes
> INFO  [] -         Total JVM Direct Memory: 378.880mb (397284478 bytes)
> INFO  [] -           Framework:             128.000mb (134217728 bytes)
> INFO  [] -           Task:                  0 bytes
> INFO  [] -           Network:               250.880mb (263066750 bytes)
> INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
> INFO  [] -     JVM Overhead:                307.200mb (322122552 bytes)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] wangyang0918 commented on a change in pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode

[GitBox]

wangyang0918 commented on a change in pull request #18119:
URL: https://github.com/apache/flink/pull/18119#discussion_r772792426



##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesSessionClusterEntrypoint.java
##
@@ -36,13 +42,48 @@ public KubernetesSessionClusterEntrypoint(Configuration 
configuration) {
 super(configuration);
 }
 
+private int blobPort;
+
 @Override
 protected DispatcherResourceManagerComponentFactory
 createDispatcherResourceManagerComponentFactory(Configuration 
configuration) {
 return 
DefaultDispatcherResourceManagerComponentFactory.createSessionComponentFactory(
 KubernetesResourceManagerFactory.getInstance());
 }
 
+@Override
+protected void updateBlobPort(int port) {

Review comment:
   Adding the interface `updateBlobPort()` `additional()` in 
`ClusterEntrypoint` and doing the implementation here are really not a good 
proposal. If we want to update the K8s internal/external services, then it 
could happen in `KubernetesResourceManagerDriver#initializeInternal()`.
   
   BTW, not only the session mode, we also need to support host network for 
application mode.

##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesSessionClusterEntrypoint.java
##
@@ -36,13 +42,48 @@ public KubernetesSessionClusterEntrypoint(Configuration 
configuration) {
 super(configuration);
 }
 
+private int blobPort;
+
 @Override
 protected DispatcherResourceManagerComponentFactory
 createDispatcherResourceManagerComponentFactory(Configuration 
configuration) {
 return 
DefaultDispatcherResourceManagerComponentFactory.createSessionComponentFactory(
 KubernetesResourceManagerFactory.getInstance());
 }
 
+@Override
+protected void updateBlobPort(int port) {
+this.blobPort = port;
+}
+
+@Override
+protected String getRPCPortRange(Configuration configuration) {

Review comment:
   If we overwrite the `HA_JOB_MANAGER_PORT_RANGE` and 
`JobManagerOptions.PORT` to 0  in `KubernetesEntrypointUtils#loadConfiguration` 
when host network enabled, then we do not need this change here.

##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecorator.java
##
@@ -132,9 +133,11 @@ private Pod decoratePod(Pod pod) {
 for (File file : localLogFiles) {
 data.put(file.getName(), Files.toString(file, 
StandardCharsets.UTF_8));
 }
-
-final Map propertiesMap =
-
getClusterSidePropertiesMap(kubernetesComponentConf.getFlinkConfiguration());
+Configuration configuration = 
kubernetesComponentConf.getFlinkConfiguration();

Review comment:
   Just like what I said above, updating the port could be moved in JM side.

##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesJobManagerFactory.java
##
@@ -93,6 +94,12 @@ private static Deployment createJobManagerDeployment(
 final Pod resolvedPod =
 new PodBuilder(flinkPod.getPodWithoutMainContainer())
 .editOrNewSpec()
+.withHostNetwork(

Review comment:
   I do not think we need to set the host network here.

##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##
@@ -86,6 +93,44 @@
 
 private static final YAMLMapper yamlMapper = new YAMLMapper();

Review comment:
   After all other comments are addressed, I believe we do not need so many 
inessential changes in `KubernetesUtils.java`.

##
File path: 
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkKubeClient.java
##
@@ -201,6 +201,23 @@ KubernetesLeaderElector createLeaderElector(
  */
 KubernetesPod loadPodFromTemplateFile(File podTemplateFile);
 
+/**
+ * Update InternalService targetPort to real value.
+ *
+ * @param jobManagerRpcPort Real jobManagerRpcPort
+ * @param blobServerPort Real blobServerPort
+ * @return Return a modified InternalService that can be used correctly
+ */
+KubernetesService updateInternalServicePort(int jobManagerRpcPort, int 
blobServerPort);

Review comment:
   I believe we could have a unified interface `CompletableFuture 
updateServicePort(String svcName, String portName, int targetPort)`. Then the 
implementation could also be simplified.
   
   ```
   public void updateServicePort(String svcName, String portName, String 
targetPort) {
  ... ...
   getService(clusterId)
   .ifPresent(
   service -> {
   final Service updatedService =
   new 

[jira] [Commented] (FLINK-25380) Introduce DynamicTableSink & DynamicTableSource for RabbitMQ

[RocMarshal (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462992#comment-17462992
 ] 

RocMarshal commented on FLINK-25380:


[~nicholasjiang] Thanks for the reminding.

In my limited read, it's also feasible to implement RabbitMQ sql connector on 
the new API.

If we don't rely on the new API, we can develop on the original API at present.
What do you think of it ?:)

> Introduce DynamicTableSink & DynamicTableSource for RabbitMQ
> 
>
> Key: FLINK-25380
> URL: https://issues.apache.org/jira/browse/FLINK-25380
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors/ RabbitMQ
>Reporter: RocMarshal
>Priority: Minor
>
> This feature should be arranged after 
>  # FLINK-21373
>  # FLINK-20628



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (FLINK-23403) Decrease default values for heartbeat timeout and interval

[Fei Feng (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-23403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462989#comment-17462989
 ] 

Fei Feng commented on FLINK-23403:
--

There is a risk if we decrease heartbeat's timeout and interval.
The reason is there will be a lots of rpc on jobmanager when a big scale job 
want to run, this will make jobmanager very busy , even can not handle heatbeat 
in time.
The solution I think , is making timeout and interval with different value when 
job is in different status, when rpc pressure is heavy (deploy task/recover 
job), increase the timeout and interval , and when rpc pressure is light (job 
is running) , decrease the interval and timeout. 
BTW, it's just a half-baked idea :D

> Decrease default values for heartbeat timeout and interval
> --
>
> Key: FLINK-23403
> URL: https://issues.apache.org/jira/browse/FLINK-23403
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: pull-request-available, stale-assigned
> Fix For: 1.15.0
>
>
> In order to speed up failure detection I suggest to decrease the default 
> values for the heartbeat timeout and interval from 50s/10s to 15s/3s.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   * 5c3e94d6e67f9b784bddb8770bbc2b519a0ec6d3 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28412)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   * 5c3e94d6e67f9b784bddb8770bbc2b519a0ec6d3 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   * 5c3e94d6e67f9b784bddb8770bbc2b519a0ec6d3 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   * 5c3e94d6e67f9b784bddb8770bbc2b519a0ec6d3 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] leonardBang commented on a change in pull request #18058: [FLINK-24571][connectors/elasticsearch] Support generating dynamic index based on proctime

[GitBox]

leonardBang commented on a change in pull request #18058:
URL: https://github.com/apache/flink/pull/18058#discussion_r772819208



##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -176,6 +207,8 @@ private static DynamicFormatter createFormatFunction(
 private static final Pattern dynamicIndexPattern = 
Pattern.compile("\\{[^\\{\\}]+\\}?");
 private static final Pattern dynamicIndexTimeExtractPattern =
 Pattern.compile(".*\\{.+\\|.*\\}.*");
+private static final Pattern dynamicIndexProcTimeExtractPattern =
+Pattern.compile(".*\\{now\\(\\)\\|.*\\}.*");

Review comment:
   I think following functions should also works, right?
   `now()` 
   `NOW()` 
`current_timestamp`
`CURRENT_TIMESTAMP`
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] leonardBang commented on a change in pull request #18058: [FLINK-24571][connectors/elasticsearch] Support generating dynamic index based on proctime

[GitBox]

leonardBang commented on a change in pull request #18058:
URL: https://github.com/apache/flink/pull/18058#discussion_r772819208



##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -176,6 +207,8 @@ private static DynamicFormatter createFormatFunction(
 private static final Pattern dynamicIndexPattern = 
Pattern.compile("\\{[^\\{\\}]+\\}?");
 private static final Pattern dynamicIndexTimeExtractPattern =
 Pattern.compile(".*\\{.+\\|.*\\}.*");
+private static final Pattern dynamicIndexProcTimeExtractPattern =
+Pattern.compile(".*\\{now\\(\\)\\|.*\\}.*");

Review comment:
   I think following function should also works, right?
   `now()` 
   `NOW()` 
`current_timestamp`
`CURRENT_TIMESTAMP`
   

##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -240,6 +273,11 @@ boolean checkIsDynamicIndexWithFormat(String index) {
 return dynamicIndexTimeExtractPattern.matcher(index).matches();
 }
 
+/** Check generate a dynamic index is based on processing time. */

Review comment:
   ```suggestion
   /** Check generate dynamic index is from system time or not. */
   ```

##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -240,6 +273,11 @@ boolean checkIsDynamicIndexWithFormat(String index) {
 return dynamicIndexTimeExtractPattern.matcher(index).matches();
 }
 
+/** Check generate a dynamic index is based on processing time. */
+boolean checkIsDynamicIndexWithProcTimeFormat(String index) {

Review comment:
   Hi, @hackergin @alpreu @matriv , Thanks for your nice work.
   I want to talk more about the diff between `proctime()` and `now()`, they 
have different return type as well as the meaning in Flink SQL, the first one 
is `TIMESTAMP_LTZ *PROCTIME*`， the later one is `TIMESTAMP_LTZ(3)`, I think the 
PR description is improper with our current implementation. I suggest changing 
the issue/PR description to that we supports a system time function(`now()` and 
`current_timestamp`)  in index pattern, this should be more clear and won't 
mislead users. WDYT?

##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -63,30 +65,59 @@
 private IndexGeneratorFactory() {}
 
 public static IndexGenerator createIndexGenerator(
-String index, List fieldNames, List dataTypes) {
+String index,
+List fieldNames,
+List dataTypes,
+ZoneId localTimeZoneId) {
 final IndexHelper indexHelper = new IndexHelper();
 if (indexHelper.checkIsDynamicIndex(index)) {
 return createRuntimeIndexGenerator(
 index,
 fieldNames.toArray(new String[0]),
 dataTypes.toArray(new DataType[0]),
-indexHelper);
+indexHelper,
+localTimeZoneId);
 } else {
 return new StaticIndexGenerator(index);
 }
 }
 
+public static IndexGenerator createIndexGenerator(
+String index, List fieldNames, List dataTypes) {
+return createIndexGenerator(index, fieldNames, dataTypes, 
ZoneId.systemDefault());
+}
+
 interface DynamicFormatter extends Serializable {
 String format(@Nonnull Object fieldValue, DateTimeFormatter formatter);
 }
 
 private static IndexGenerator createRuntimeIndexGenerator(
-String index, String[] fieldNames, DataType[] fieldTypes, 
IndexHelper indexHelper) {
+String index,
+String[] fieldNames,
+DataType[] fieldTypes,
+IndexHelper indexHelper,
+ZoneId localTimeZoneId) {
 final String dynamicIndexPatternStr = 
indexHelper.extractDynamicIndexPatternStr(index);
 final String indexPrefix = index.substring(0, 
index.indexOf(dynamicIndexPatternStr));
 final String indexSuffix =
 index.substring(indexPrefix.length() + 
dynamicIndexPatternStr.length());
 
+final boolean isDynamicIndexWithProcTimeFormat =
+indexHelper.checkIsDynamicIndexWithProcTimeFormat(index);
+if (isDynamicIndexWithProcTimeFormat) {

Review comment:
   ```suggestion
   if (indexHelper.checkIsDynamicIndexWithProcTimeFormat(index)) {
   ```

##
File path: 
flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##
@@ -240,6 

[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread

[GitBox]

flinkbot edited a comment on pull request #17988:
URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654


   
   ## CI report:
   
   * f015c1d120d1b63a3014126ff28b07cff58fc9c0 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28039)
 
   * e7be85162e0b431d518ab1ffe8b59283338d00b7 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28411)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread

[GitBox]

flinkbot edited a comment on pull request #17988:
URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654


   
   ## CI report:
   
   * f015c1d120d1b63a3014126ff28b07cff58fc9c0 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28039)
 
   * e7be85162e0b431d518ab1ffe8b59283338d00b7 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-25397) support grouped_execution

[ZhuoYu Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhuoYu Chen updated FLINK-25397:

Description: 
Performing data bucketing execution: two tables (orders, orders_item), divided 
into buckets (bucketing) based on the same fields (orderid) and the same number 
of buckets. In join by order id, join and aggregation calculations can be 
performed independently, because the same order ids of both tables are divided 
into buckets with the same ids.
This has several advantages.：
1. Whenever a bucket of data is computed, the memory occupied by this bucket 
can be released immediately, so memory consumption can be limited by 
controlling the number of buckets processed in parallel.
2. reduces a lot of shuffling

  was:
Performing data bucketing execution: two tables (orders, orders_item), divided 
into buckets (bucketing) based on the same fields (orderid) and the same number 
of buckets. In join by order id, join and aggregation calculations can be 
performed independently, because the same order ids of both tables are divided 
into buckets with the same ids.
This has several advantages. 1. Whenever a bucket of data is computed, the 
memory occupied by this bucket can be released immediately, so memory 
consumption can be limited by controlling the number of buckets processed in 
parallel.
2. reduces a lot of shuffling


> support grouped_execution
> -
>
> Key: FLINK-25397
> URL: https://issues.apache.org/jira/browse/FLINK-25397
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Legacy Planner, Table SQL / Planner, Table 
> SQL / Runtime
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Major
>
> Performing data bucketing execution: two tables (orders, orders_item), 
> divided into buckets (bucketing) based on the same fields (orderid) and the 
> same number of buckets. In join by order id, join and aggregation 
> calculations can be performed independently, because the same order ids of 
> both tables are divided into buckets with the same ids.
> This has several advantages.：
> 1. Whenever a bucket of data is computed, the memory occupied by this bucket 
> can be released immediately, so memory consumption can be limited by 
> controlling the number of buckets processed in parallel.
> 2. reduces a lot of shuffling



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (FLINK-25397) support grouped_execution

[ZhuoYu Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhuoYu Chen updated FLINK-25397:

Summary: support grouped_execution  (was: grouped_execution)

> support grouped_execution
> -
>
> Key: FLINK-25397
> URL: https://issues.apache.org/jira/browse/FLINK-25397
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Legacy Planner, Table SQL / Planner, Table 
> SQL / Runtime
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Major
>
> Performing data bucketing execution: two tables (orders, orders_item), 
> divided into buckets (bucketing) based on the same fields (orderid) and the 
> same number of buckets. In join by order id, join and aggregation 
> calculations can be performed independently, because the same order ids of 
> both tables are divided into buckets with the same ids.
> This has several advantages. 1. Whenever a bucket of data is computed, the 
> memory occupied by this bucket can be released immediately, so memory 
> consumption can be limited by controlling the number of buckets processed in 
> parallel.
> 2. reduces a lot of shuffling



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (FLINK-25397) support grouped_execution

[ZhuoYu Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhuoYu Chen updated FLINK-25397:

Description: 
Performing data bucketing execution: two tables (orders, orders_item), divided 
into buckets (bucketing) based on the same fields (orderid) and the same number 
of buckets. In join by order id, join and aggregation calculations can be 
performed independently, because the same order ids of both tables are divided 
into buckets with the same ids.
This has several advantages：
1. Whenever a bucket of data is computed, the memory occupied by this bucket 
can be released immediately, so memory consumption can be limited by 
controlling the number of buckets processed in parallel.
2. reduces a lot of shuffling

  was:
Performing data bucketing execution: two tables (orders, orders_item), divided 
into buckets (bucketing) based on the same fields (orderid) and the same number 
of buckets. In join by order id, join and aggregation calculations can be 
performed independently, because the same order ids of both tables are divided 
into buckets with the same ids.
This has several advantages.：
1. Whenever a bucket of data is computed, the memory occupied by this bucket 
can be released immediately, so memory consumption can be limited by 
controlling the number of buckets processed in parallel.
2. reduces a lot of shuffling


> support grouped_execution
> -
>
> Key: FLINK-25397
> URL: https://issues.apache.org/jira/browse/FLINK-25397
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Legacy Planner, Table SQL / Planner, Table 
> SQL / Runtime
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Major
>
> Performing data bucketing execution: two tables (orders, orders_item), 
> divided into buckets (bucketing) based on the same fields (orderid) and the 
> same number of buckets. In join by order id, join and aggregation 
> calculations can be performed independently, because the same order ids of 
> both tables are divided into buckets with the same ids.
> This has several advantages：
> 1. Whenever a bucket of data is computed, the memory occupied by this bucket 
> can be released immediately, so memory consumption can be limited by 
> controlling the number of buckets processed in parallel.
> 2. reduces a lot of shuffling



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25397) grouped_execution

[ZhuoYu Chen (Jira)]

ZhuoYu Chen created FLINK-25397:
---

 Summary: grouped_execution
 Key: FLINK-25397
 URL: https://issues.apache.org/jira/browse/FLINK-25397
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Legacy Planner, Table SQL / Planner, Table 
SQL / Runtime
Affects Versions: 1.15.0
Reporter: ZhuoYu Chen


Performing data bucketing execution: two tables (orders, orders_item), divided 
into buckets (bucketing) based on the same fields (orderid) and the same number 
of buckets. In join by order id, join and aggregation calculations can be 
performed independently, because the same order ids of both tables are divided 
into buckets with the same ids.
This has several advantages. 1. Whenever a bucket of data is computed, the 
memory occupied by this bucket can be released immediately, so memory 
consumption can be limited by controlling the number of buckets processed in 
parallel.
2. reduces a lot of shuffling



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28410)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zuston commented on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

zuston commented on pull request #18157:
URL: https://github.com/apache/flink/pull/18157#issuecomment-998442727


   @Myasuka 
   @KarmaGYZ 
   
   Could you help give me some advice to solve this problem. We also meet it in 
our production practise.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-25321) standalone deploy on k8s,pod always OOM killed,actual heap memory usage is normal, gc is normal

[Gao Fei (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462965#comment-17462965
 ] 

Gao Fei commented on FLINK-25321:
-

[~wangyang0918] OK，thanks

> standalone deploy on k8s,pod always OOM killed,actual heap memory usage is 
> normal, gc is normal
> ---
>
> Key: FLINK-25321
> URL: https://issues.apache.org/jira/browse/FLINK-25321
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.11.3
> Environment: Flink 1.11.3
> k8s v1.21.0
> standlone deployment
>Reporter: Gao Fei
>Priority: Major
>
> Start a cluster on k8s, deploy in standalone mode, a jobmanager pod (1G) and 
> a taskmanager pod (3372MB limit), the total memory configuration of the Flink 
> TM process is 3072MB, and the managed configuration is 0, both of which are 
> on the heap memory. Now the pod It will always be OOM killed, and the total 
> process memory will always exceed 3072MB. I saw that the system has adopted 
> jemlloc. There is no 64M problem. The application itself has not applied for 
> direct memory. It is strange why the process is always killed by OOM after a 
> period of time.
>  
> INFO  [] - Final TaskExecutor Memory configuration:
> INFO  [] -   Total Process Memory:          3.000gb (3221225472 bytes)
> INFO  [] -     Total Flink Memory:          2.450gb (2630667464 bytes)
> INFO  [] -       Total JVM Heap Memory:     2.080gb (2233382986 bytes)
> INFO  [] -         Framework:               128.000mb (134217728 bytes)
> INFO  [] -         Task:                    1.955gb (2099165258 bytes)
> INFO  [] -       Total Off-heap Memory:     378.880mb (397284478 bytes)
> INFO  [] -         Managed:                 0 bytes
> INFO  [] -         Total JVM Direct Memory: 378.880mb (397284478 bytes)
> INFO  [] -           Framework:             128.000mb (134217728 bytes)
> INFO  [] -           Task:                  0 bytes
> INFO  [] -           Network:               250.880mb (263066750 bytes)
> INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
> INFO  [] -     JVM Overhead:                307.200mb (322122552 bytes)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #18007: [FLINK-25085][runtime] Add a scheduled thread pool in Endpoint and close it when the endpoint is stopped

[GitBox]

flinkbot edited a comment on pull request #18007:
URL: https://github.com/apache/flink/pull/18007#issuecomment-985453331


   
   ## CI report:
   
   * 9edd3e9c9296c1d8610a2315e44a381b0845deef Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28368)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] liyubin117 commented on pull request #18146: [FLINK-25369][table] CatalogManager provide right tables or views in given catalog database

[GitBox]

liyubin117 commented on pull request #18146:
URL: https://github.com/apache/flink/pull/18146#issuecomment-998440668


   @RocMarshal CI result has been success now. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-23687) Add Sql query hint to enable LookupJoin shuffle by join key of left input

[HunterHunter (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-23687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462963#comment-17462963
 ] 

HunterHunter commented on FLINK-23687:
--

Well, looking forward to your contribution, I will continue to pay attention

> Add Sql query hint to enable LookupJoin shuffle by join key of left input
> -
>
> Key: FLINK-23687
> URL: https://issues.apache.org/jira/browse/FLINK-23687
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jing Zhang
>Assignee: Jing Zhang
>Priority: Major
>
> Add Sql query hint to enable LookupJoin shuffle by join key of left input



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink-ml] yunfengzhou-hub commented on pull request #42: [FLINK-25394] Upgrade log4j to 2.17.0 to address CVE-2021-45105

[GitBox]

yunfengzhou-hub commented on pull request #42:
URL: https://github.com/apache/flink-ml/pull/42#issuecomment-998439869


   Hi @Abdelrahman-IK , Thanks for the PR. It looks good to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-25262) Support to send data to lookup table for KeyGroupStreamPartitioner way for SQL

[Jing Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhang closed FLINK-25262.
--
Resolution: Duplicate

> Support to send data to  lookup table for KeyGroupStreamPartitioner way for 
> SQL
> ---
>
> Key: FLINK-25262
> URL: https://issues.apache.org/jira/browse/FLINK-25262
> Project: Flink
>  Issue Type: Improvement
>Reporter: hehuiyuan
>Priority: Minor
> Attachments: image-2021-12-12-15-15-48-540.png, 
> image-2021-12-12-15-18-08-574.png
>
>
> Send data to lookup table  by hash , which  can improve cache hit rate in 
> lookup table, futher improve processing performance and reduce the size of 
> cache.
>  
> Shoulder we consider to introducing it?
>  
>  
> !image-2021-12-12-15-18-08-574.png|width=419,height=193!
>  
> I have a simple test.  The parallelism is 10 and the kafka source has 100 
> million records and the hbase lookuptable has 100 thousands records. It need 
> 100 minutes for forward and 5 minutes for hash.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (FLINK-25380) Introduce DynamicTableSink & DynamicTableSource for RabbitMQ

[Nicholas Jiang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462959#comment-17462959
 ] 

Nicholas Jiang commented on FLINK-25380:


[~RocMarshal], IMO, the RabbitMQ sql connector could only integrates with the 
legacy RMQSource and RMQSink. The mentioned JIRAs are based on the new Source 
and Sink API. Therefore this issue is unnecessary to link above JIRAs.

> Introduce DynamicTableSink & DynamicTableSource for RabbitMQ
> 
>
> Key: FLINK-25380
> URL: https://issues.apache.org/jira/browse/FLINK-25380
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors/ RabbitMQ
>Reporter: RocMarshal
>Priority: Minor
>
> This feature should be arranged after 
>  # FLINK-21373
>  # FLINK-20628



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (FLINK-23687) Add Sql query hint to enable LookupJoin shuffle by join key of left input

[Jing Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-23687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462958#comment-17462958
 ] 

Jing Zhang commented on FLINK-23687:


Also invite [~ykt836][~twalthr][~jark][~godfreyhe], WDYT?

> Add Sql query hint to enable LookupJoin shuffle by join key of left input
> -
>
> Key: FLINK-23687
> URL: https://issues.apache.org/jira/browse/FLINK-23687
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jing Zhang
>Assignee: Jing Zhang
>Priority: Major
>
> Add Sql query hint to enable LookupJoin shuffle by join key of left input



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

flinkbot edited a comment on pull request #18157:
URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589


   
   ## CI report:
   
   * 0f1c86c86ba5a4c3388ef2aac95af103a3bfcced Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28409)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-23687) Add Sql query hint to enable LookupJoin shuffle by join key of left input

[Jing Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-23687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462956#comment-17462956
 ] 

Jing Zhang edited comment on FLINK-23687 at 12/21/21, 3:16 AM:
---

Hi, [~HunterHunter] Thanks for attention.
Since look up join is commonly used, there are many optimization requirements 
on this topic. Such as:
1. Enable partitioned join to raise cache hint ratio.
2. Solve the data skew case in the first point
3. Enable miniBatch optimization to reduce RPC call.
We have finished the those points in the internal version already and validate 
them in production jobs. 
We are planning to push all of the optimization to the community.
Currently, we have discussed this topic. 
Including hear requirements from more users,  designing SQL syntax, API update 
and implement update.
Maybe would create a FLIP after we have achieved same goals and send a discuss 
in user and dev mail list.
Welcome to join the discuss.


was (Author: qingru zhang):
Hi, [~HunterHunter] Thanks for attention.

> Add Sql query hint to enable LookupJoin shuffle by join key of left input
> -
>
> Key: FLINK-23687
> URL: https://issues.apache.org/jira/browse/FLINK-23687
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jing Zhang
>Assignee: Jing Zhang
>Priority: Major
>
> Add Sql query hint to enable LookupJoin shuffle by join key of left input



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot commented on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

flinkbot commented on pull request #18157:
URL: https://github.com/apache/flink/pull/18157#issuecomment-998436720


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 0f1c86c86ba5a4c3388ef2aac95af103a3bfcced (Tue Dec 21 
03:16:58 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-17808).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

flinkbot commented on pull request #18157:
URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589


   
   ## CI report:
   
   * 0f1c86c86ba5a4c3388ef2aac95af103a3bfcced UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-17808) Rename checkpoint meta file to "_metadata" until it has completed writing

[Junfan Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462957#comment-17462957
 ] 

Junfan Zhang commented on FLINK-17808:
--

Could i take over this ticket? [~yunta] [~guoyangze] [~zhoujira86]

Draft PR link: https://github.com/apache/flink/pull/18157

If OK, i will optimize and add more tests on it.

> Rename checkpoint meta file to "_metadata" until it has completed writing
> -
>
> Key: FLINK-17808
> URL: https://issues.apache.org/jira/browse/FLINK-17808
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Priority: Minor
>  Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
>
> In practice, some developers or customers would use some strategy to find the 
> recent _metadata as the checkpoint to recover (e.g as many proposals in 
> FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean 
> the checkpoint have been completed as the writing to create the "_meatadata" 
> file could break as some force quit (e.g. yarn application -kill).
> We could create the checkpoint meta stream to write data to file named as 
> "_metadata.inprogress" and renamed it to "_metadata" once completed writing. 
> By doing so, we could ensure the "_metadata" is not broken.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (FLINK-17808) Rename checkpoint meta file to "_metadata" until it has completed writing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-17808:
---
Labels: auto-deprioritized-major pull-request-available  (was: 
auto-deprioritized-major)

> Rename checkpoint meta file to "_metadata" until it has completed writing
> -
>
> Key: FLINK-17808
> URL: https://issues.apache.org/jira/browse/FLINK-17808
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Priority: Minor
>  Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
>
> In practice, some developers or customers would use some strategy to find the 
> recent _metadata as the checkpoint to recover (e.g as many proposals in 
> FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean 
> the checkpoint have been completed as the writing to create the "_meatadata" 
> file could break as some force quit (e.g. yarn application -kill).
> We could create the checkpoint meta stream to write data to file named as 
> "_metadata.inprogress" and renamed it to "_metadata" once completed writing. 
> By doing so, we could ensure the "_metadata" is not broken.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [flink] flinkbot edited a comment on pull request #18146: [FLINK-25369][table] CatalogManager provide right tables or views in given catalog database

[GitBox]

flinkbot edited a comment on pull request #18146:
URL: https://github.com/apache/flink/pull/18146#issuecomment-997153629


   
   ## CI report:
   
   * c87817ded7f92df4066277a0e944563315541c22 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28378)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zuston opened a new pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…

[GitBox]

zuston opened a new pull request #18157:
URL: https://github.com/apache/flink/pull/18157


   … completed writing
   
   ## What is the purpose of the change
   
   In order to solve the situation where the checkpoint metadata file is 
suspended from being written under certain abnormal conditions, but it is still 
visible to the outside. However the metadata file at this time is invalid.
   
   ## Brief change log
   
   Using the `RecoverableFsDataOutputStream` close and commit to ensure the 
writing file atomicity. 
   
   ## Verifying this change
   
   
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't 
know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…

[GitBox]

flinkbot edited a comment on pull request #17521:
URL: https://github.com/apache/flink/pull/17521#issuecomment-946651087


   
   ## CI report:
   
   * cdf7db8a02f2962fc52221d2782b6aa4099c6d24 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28388)
 
   * 1484f8aada73558e39ba5fdcff5373f3ebf66aee Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28407)
 
   * 5b853455f1d2980fd2bb0b86f845637e478735a4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on pull request #18058: [FLINK-24571][connectors/elasticsearch] Support generating dynamic index based on proctime

[GitBox]

wuchong commented on pull request #18058:
URL: https://github.com/apache/flink/pull/18058#issuecomment-998433028


   @leonardBang , do you want to review and merge this PR? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-23687) Add Sql query hint to enable LookupJoin shuffle by join key of left input

[Jing Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-23687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462956#comment-17462956
 ] 

Jing Zhang edited comment on FLINK-23687 at 12/21/21, 3:08 AM:
---

Hi, [~HunterHunter] Thanks for attention.


was (Author: qingru zhang):
Hi, [~HunterHunter] Thanks for attention.

> Add Sql query hint to enable LookupJoin shuffle by join key of left input
> -
>
> Key: FLINK-23687
> URL: https://issues.apache.org/jira/browse/FLINK-23687
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Jing Zhang
>Assignee: Jing Zhang
>Priority: Major
>
> Add Sql query hint to enable LookupJoin shuffle by join key of left input



--
This message was sent by Atlassian Jira
(v8.20.1#820001)