[GitHub] [flink] link3280 commented on pull request #21292: [FLINK-28617][SQL Gateway] Support stop job statement in SqlGatewayService
link3280 commented on PR #21292: URL: https://github.com/apache/flink/pull/21292#issuecomment-1312396064 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
SteNicholas commented on code in PR #357: URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020706975 ## flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java: ## @@ -226,6 +227,13 @@ public void createTable(ObjectPath tablePath, UpdateSchema updateSchema, boolean e); } Table table = newHmsTable(tablePath); + +if (hiveConf.getEnum(TABLE_TYPE.key(), TableType.MANAGED_TABLE) Review Comment: It's better to move this logic to `newHmsTable` which initialzes the `Table` and sets the parameters of the `Table`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020706445
##
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##
@@ -226,6 +227,13 @@ public void createTable(ObjectPath tablePath, UpdateSchema
updateSchema, boolean
e);
}
Table table = newHmsTable(tablePath);
+
+if (hiveConf.getEnum(TABLE_TYPE.key(), TableType.MANAGED_TABLE)
+.equals(TableType.EXTERNAL_TABLE)) {
+table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
```suggestion
table.setTableType(TableType.EXTERNAL_TABLE.name());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #21281: [FLINK-29969][checkpoint] Show the root cause when exceeded checkpoint tolerable failure threshold
1996fanrui commented on PR #21281: URL: https://github.com/apache/flink/pull/21281#issuecomment-1312386445 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21303: [FLINK-30002][checkpoint] Change the alignmentTimeout to alignedCheckpointTimeout
flinkbot commented on PR #21303: URL: https://github.com/apache/flink/pull/21303#issuecomment-1312358601 ## CI report: * 86033ee0118d054f5ebd93eb40e2b0488cfb4608 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30002) Change the alignmentTimeout to alignedCheckpointTimeout
[ https://issues.apache.org/jira/browse/FLINK-30002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30002: --- Labels: pull-request-available (was: ) > Change the alignmentTimeout to alignedCheckpointTimeout > --- > > Key: FLINK-30002 > URL: https://issues.apache.org/jira/browse/FLINK-30002 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.16.0 >Reporter: Rui Fan >Priority: Minor > Labels: pull-request-available > Fix For: 1.17.0 > > > The alignmentTimeout has been changed to alignedCheckpointTimeout in > FLINK-23041 . > But some fields or methods still use alignmentTimeout. They should be renamed > to alignedCheckpointTimeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] 1996fanrui opened a new pull request, #21303: [FLINK-30002][checkpoint] Change the alignmentTimeout to alignedCheckpointTimeout
1996fanrui opened a new pull request, #21303: URL: https://github.com/apache/flink/pull/21303 ## What is the purpose of the change The alignmentTimeout has been changed to alignedCheckpointTimeout in #16227 [FLINK-23041](https://issues.apache.org/jira/browse/FLINK-23041) . But some fields or methods still use alignmentTimeout. They should be renamed to alignedCheckpointTimeout. ## Brief change log Change the alignmentTimeout to alignedCheckpointTimeout ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not documented -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-30002) Change the alignmentTimeout to alignedCheckpointTimeout
Rui Fan created FLINK-30002: --- Summary: Change the alignmentTimeout to alignedCheckpointTimeout Key: FLINK-30002 URL: https://issues.apache.org/jira/browse/FLINK-30002 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing Affects Versions: 1.16.0 Reporter: Rui Fan Fix For: 1.17.0 The alignmentTimeout has been changed to alignedCheckpointTimeout in FLINK-23041 . But some fields or methods still use alignmentTimeout. They should be renamed to alignedCheckpointTimeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #21302: [FLINK-29992][hive] fix lookup join fail with Hive table as lookup table source
flinkbot commented on PR #21302: URL: https://github.com/apache/flink/pull/21302#issuecomment-1312349043 ## CI report: * 0d8107eee665c99de06782684d177b22bf312b25 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29992) Join execution plan parsing error
[
https://issues.apache.org/jira/browse/FLINK-29992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29992:
---
Labels: pull-request-available (was: )
> Join execution plan parsing error
> -
>
> Key: FLINK-29992
> URL: https://issues.apache.org/jira/browse/FLINK-29992
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: HunterXHunter
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> //
> tableEnv.executeSql(" CREATE CATALOG hive WITH (\n"
> + " 'type' = 'hive',\n"
> + " 'default-database' = 'flinkdebug',\n"
> + " 'hive-conf-dir' = '/programe/hadoop/hive-3.1.2/conf'\n"
> + " )");
> tableEnv.executeSql("create table datagen_tbl (\n"
> + "id STRING\n"
> + ",name STRING\n"
> + ",age bigint\n"
> + ",ts bigint\n"
> + ",`par` STRING\n"
> + ",pro_time as PROCTIME()\n"
> + ") with (\n"
> + " 'connector'='datagen'\n"
> + ",'rows-per-second'='10'\n"
> + " \n"
> + ")");
> String dml1 = "select * "
> + " from datagen_tbl as p "
> + " join hive.flinkdebug.default_hive_src_tbl "
> + " FOR SYSTEM_TIME AS OF p.pro_time AS c"
> + " ON p.id = c.id";
> // Execution succeeded
> System.out.println(tableEnv.explainSql(dml1));
> String dml2 = "select p.id "
> + " from datagen_tbl as p "
> + " join hive.flinkdebug.default_hive_src_tbl "
> + " FOR SYSTEM_TIME AS OF p.pro_time AS c"
> + " ON p.id = c.id";
> // Throw an exception
> System.out.println(tableEnv.explainSql(dml2)); {code}
> {code:java}
> org.apache.flink.table.api.TableException: Cannot generate a valid execution
> plan for the given query: FlinkLogicalCalc(select=[id]) +-
> FlinkLogicalJoin(condition=[=($0, $1)], joinType=[inner]) :-
> FlinkLogicalCalc(select=[id]) : +-
> FlinkLogicalTableSourceScan(table=[[default_catalog, default_database,
> datagen_tbl]], fields=[id, name, age, ts, par]) +-
> FlinkLogicalSnapshot(period=[$cor1.pro_time]) +-
> FlinkLogicalTableSourceScan(table=[[hive, flinkdebug, default_hive_src_tbl,
> project=[id]]], fields=[id])This exception indicates that the query uses an
> unsupported SQL feature. Please check the documentation for the set of
> currently supported SQL features. at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:70)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] luoyuxia opened a new pull request, #21302: [FLINK-29992][hive] fix lookup join fail with Hive table as lookup table source
luoyuxia opened a new pull request, #21302: URL: https://github.com/apache/flink/pull/21302 ## What is the purpose of the change To fix the lookup join fail with Hive table as lookup table source ## Brief change log Implement `copy` method in `HiveLookupTableSource`. ## Verifying this change Added test `HiveLookupJoinITCase` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29992) Join execution plan parsing error
[
https://issues.apache.org/jira/browse/FLINK-29992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632594#comment-17632594
]
luoyuxia commented on FLINK-29992:
--
FLINK-29138 introduce project push down for lookup source, then the
`PushProjectIntoTableSourceScanRule` will work which will then
{code:java}
sourceTable.tableSource().copy(); {code}
But the HiveLookupTableSource doesn't implement copy method which will
delegate to it's parent method, and then copy a `HiveTableSource` which is not
a lookup source.
> Join execution plan parsing error
> -
>
> Key: FLINK-29992
> URL: https://issues.apache.org/jira/browse/FLINK-29992
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: HunterXHunter
>Priority: Major
>
> {code:java}
> //
> tableEnv.executeSql(" CREATE CATALOG hive WITH (\n"
> + " 'type' = 'hive',\n"
> + " 'default-database' = 'flinkdebug',\n"
> + " 'hive-conf-dir' = '/programe/hadoop/hive-3.1.2/conf'\n"
> + " )");
> tableEnv.executeSql("create table datagen_tbl (\n"
> + "id STRING\n"
> + ",name STRING\n"
> + ",age bigint\n"
> + ",ts bigint\n"
> + ",`par` STRING\n"
> + ",pro_time as PROCTIME()\n"
> + ") with (\n"
> + " 'connector'='datagen'\n"
> + ",'rows-per-second'='10'\n"
> + " \n"
> + ")");
> String dml1 = "select * "
> + " from datagen_tbl as p "
> + " join hive.flinkdebug.default_hive_src_tbl "
> + " FOR SYSTEM_TIME AS OF p.pro_time AS c"
> + " ON p.id = c.id";
> // Execution succeeded
> System.out.println(tableEnv.explainSql(dml1));
> String dml2 = "select p.id "
> + " from datagen_tbl as p "
> + " join hive.flinkdebug.default_hive_src_tbl "
> + " FOR SYSTEM_TIME AS OF p.pro_time AS c"
> + " ON p.id = c.id";
> // Throw an exception
> System.out.println(tableEnv.explainSql(dml2)); {code}
> {code:java}
> org.apache.flink.table.api.TableException: Cannot generate a valid execution
> plan for the given query: FlinkLogicalCalc(select=[id]) +-
> FlinkLogicalJoin(condition=[=($0, $1)], joinType=[inner]) :-
> FlinkLogicalCalc(select=[id]) : +-
> FlinkLogicalTableSourceScan(table=[[default_catalog, default_database,
> datagen_tbl]], fields=[id, name, age, ts, par]) +-
> FlinkLogicalSnapshot(period=[$cor1.pro_time]) +-
> FlinkLogicalTableSourceScan(table=[[hive, flinkdebug, default_hive_src_tbl,
> project=[id]]], fields=[id])This exception indicates that the query uses an
> unsupported SQL feature. Please check the documentation for the set of
> currently supported SQL features. at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:70)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30001) sql-client.sh start failed
xiaohang.li created FLINK-30001: --- Summary: sql-client.sh start failed Key: FLINK-30001 URL: https://issues.apache.org/jira/browse/FLINK-30001 Project: Flink Issue Type: Bug Components: Command Line Client Affects Versions: 1.15.2, 1.16.0 Reporter: xiaohang.li [hadoop@master flink-1.15.0]$ ./bin/sql-client.sh Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR or HADOOP_CLASSPATH was set. Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set. Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue. at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201) at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:163) at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:111) at org.apache.flink.table.client.gateway.context.ExecutionContext.(ExecutionContext.java:66) at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:247) at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) ... 1 more Caused by: org.apache.flink.table.api.TableException: Unexpected error when trying to load service provider for factories. at org.apache.flink.table.factories.FactoryUtil.lambda$discoverFactories$19(FactoryUtil.java:813) at java.util.ArrayList.forEach(ArrayList.java:1259) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:799) at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:517) at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:154) ... 8 more Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.planner.loader.DelegateExecutorFactory could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.flink.table.factories.ServiceLoaderUtil.load(ServiceLoaderUtil.java:42) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:798) ... 10 more Caused by: java.lang.ExceptionInInitializerError at org.apache.flink.table.planner.loader.PlannerModule.getInstance(PlannerModule.java:135) at org.apache.flink.table.planner.loader.DelegateExecutorFactory.(DelegateExecutorFactory.java:34) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 14 more Caused by: org.apache.flink.table.api.TableException: Could not initialize the table planner components loader. at org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:123) at org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:52) at org.apache.flink.table.planner.loader.PlannerModule$PlannerComponentsHolder.(PlannerModule.java:131) ... 22 more Caused by: java.nio.file.FileAlreadyExistsException: /tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at
[GitHub] [flink] 1996fanrui commented on pull request #21193: [hotfix] Add the final and fix typo
1996fanrui commented on PR #21193: URL: https://github.com/apache/flink/pull/21193#issuecomment-1312333280 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29928) Allow sharing (RocksDB) memory between slots
[
https://issues.apache.org/jira/browse/FLINK-29928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-29928:
--
Description:
h1. Background and motivation
RocksDB is one of the main consumers of off-heap memory, which it uses for
BlockCache, MemTables, Indices and Bloom Filters.
Since 1.10 (FLINK-7289), it is possible to:
- share these objects among RocksDB instances of the same slot
- bound the total memory usage by all RocksDB instances of a TM
The memory is divided between the slots equally (unless using fine-grained
resource control).
This is sub-optimal if some slots contain more memory intensive tasks than the
others.
The proposal is to widen the scope of sharing memory to TM, so that it can be
shared across all of its RocksDB instances.
That would reduce the overall memory consumption in exchange for resource
isolation.
h1. Proposed changes
h2. Configuration
- introduce "taskmanager.memory.managed.shared-fraction" (0..1, default 0)
-- cluster-level (yaml only)
-- the non-shared memory will be used as it is now (exclusively per-slot)
- -introduce "state.backend.memory.share-scope"-
-- -job-level (yaml and StateBackend)-
-- -possible values: NONE, SLOT, TASK_MANAGER-
-- -default: not set-
-- override "state.backend.rocksdb.memory.fixed-per-slot" if shared-fraction >
0 (but don't deprecate it, because it specifies the size)
- rely on the existing "state.backend.rocksdb.memory.managed" to decide
whether the shared memory is managed or unmanaged
- when calculating TM-wise shared memory, ignore
"taskmanager.memory.managed.consumer-weights" because RocksDB is the only
consumer so far
- similarly, exclude StateBackend from weights calculations, so other
consumers (e.g. PYTHON) can better utilize exclusive slot memory
- use cluster-level or default configuration when creating TM-wise shared
RocksDB objects, e.g. "state.backend.rocksdb.memory.managed",
"state.backend.rocksdb.memory.write-buffer-ratio"
h2. Example
{code:java}
taskmanager.memory.managed.size: 1gb
taskmanager.memory.managed.shared-fraction: .75 # all slots share 750Mb of
shared managed memory
taskmanager.numberOfTaskSlots: 10 # each task slot gets 25Mb of
exclusive managed memory
cluster.fine-grained-resource-management.enabled: false
job 1:
state.backend.rocksdb.memory.managed: true
job 2:
state.backend.rocksdb.memory.managed: true
job 3:
state.backend.rocksdb.memory.fixed-per-slot: 50M # ignored
state.backend.rocksdb.memory.managed: true
job 4:
state.backend.memory.share-scope: TASK_MANAGER
state.backend.rocksdb.memory.managed: false
job 5:
state.backend.memory.share-scope: TASK_MANAGER
state.backend.rocksdb.memory.managed: false
{code}
Jobs 1, 2, 3 will use the same 750Mb of managed memory and will compete with
each other.
Jobs 4 and 5 will use the same 750Mb of unmanaged memory and will compete with
each other. 750Mb here is calculated using the same settings
(managed.shared-fraction) to avoid adding additional parameters
(unmanaged.shared-fraction).
Python code (or other consumers) will be able to use up to 25mb per slot.
h2. Creating and sharing RocksDB objects
Introduce sharedMemoryManager to TaskManager.
Then, similarly to the current slot-wise sharing:
- Memory manager manages OpaqueMemoryResource
- Creation of Cache object is done from the backend code on the first call
So flink-runtime doesn't have to depend on state backend.
h2. Class loading and resolution
RocksDB state backend is already a part of the distribution.
However, if a job also includes it then classloader.resolve-order should be set
to parent-first to prevent conflicts.
h2. Lifecycle
The cache object should be destroyed on TM termnation; job or task completion
should NOT close it.
h1. Testing
* One way to test that the same RocksDB cache is used is via RocksDB metrics.
* ITCases parameterization
* manual and unit tests
h1. Limitations
- classloader.resolve-order=child-first is not supported
- fine-grained-resource-management is not supported
- only RocksDB will be able to use TM-wise shared memory; other consumers may
be adjusted later
h1. Rejected alternatives
- set total "fixed-per-slot" to a larger value, essentially overcommitting
unmanaged memory - doesn't work well in containerized environments (OOMErrors)
- set numberOfTaskSlots=1 and allow sharing the same slot between any tasks -
requires more invasive changes in scheduler and TM
cc: [~yunta], [~ym], [~liyu]
was:
h1. Background and motivation
RocksDB is one of the main consumers of off-heap memory, which it uses for
BlockCache, MemTables, Indices and Bloom Filters.
Since 1.10 (FLINK-7289), it is possible to:
- share these objects among RocksDB instances of the same slot
- bound the total memory usage by all RocksDB instances of a TM
The memory is divided between the slots equally (unless using fine-grained
resource control).
[GitHub] [flink] mxm commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission
mxm commented on PR #21023: URL: https://github.com/apache/flink/pull/21023#issuecomment-1311937328 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] mxm commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission
mxm commented on PR #21023: URL: https://github.com/apache/flink/pull/21023#issuecomment-1311934193 I've removed the dependency on #21008. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ericxiao251 commented on pull request #21077: [FLINK-29498] Add Scala Async Retry Strategies and ResultPredicates Helper Classes
ericxiao251 commented on PR #21077: URL: https://github.com/apache/flink/pull/21077#issuecomment-1311884781 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632436#comment-17632436
]
Jason Kania commented on FLINK-24302:
-
[~syhily] The Pulsar team is asking for the entire stack trace to be able to
investigate the issue further
(https://github.com/apache/pulsar/issues/17989#issuecomment-1275530307).
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: test-stability
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
> https://github.com/apache/pulsar/pull/15216, we need to add
> autoScaledReceiverQueueSizeEnabled option to enable
[GitHub] [flink] zentol closed pull request #20783: add Text
zentol closed pull request #20783: add Text URL: https://github.com/apache/flink/pull/20783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28341) Fix comment in BytesKeyNormalizationUtil.java
[
https://issues.apache.org/jira/browse/FLINK-28341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632432#comment-17632432
]
Chesnay Schepler commented on FLINK-28341:
--
master: 2454dfa3aa27006198cd969840b06fe7313b56d0
> Fix comment in BytesKeyNormalizationUtil.java
> -
>
> Key: FLINK-28341
> URL: https://issues.apache.org/jira/browse/FLINK-28341
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.15.1
>Reporter: shen
>Priority: Minor
> Labels: comment, pull-request-available, starter
> Fix For: 1.17.0
>
>
> The comment
> [here|https://github.com/apache/flink/blob/release-1.15.1-rc1/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/BytesKeyNormalizationUtil.java#L74]
> is not correct since
> [Byte.MIN_VALUE|https://docs.oracle.com/javase/7/docs/api/java/lang/Byte.html#MIN_VALUE]
> = -128,
> [Byte.MAX_VALUE|https://docs.oracle.com/javase/7/docs/api/java/lang/Byte.html#MAX_VALUE]
> = 127.
> And I think [code
> below|https://github.com/apache/flink/blob/release-1.15.1-rc1/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/BytesKeyNormalizationUtil.java#L77-L79]
> can be simplified as:
> {code:java}
> import org.junit.Assert;
> import org.junit.Test;
> public class TestIntegerConvertion {
> @Test
> public void testConvertByteInteger() {
> for (byte i = Byte.MIN_VALUE; ; ++i) {
> Assert.assertEquals(convertByFlink(i), convertSimplified(i));
> if (i == Byte.MAX_VALUE)
> break;
> }
> }
> private byte convertByFlink(byte originValue) {
> int highByte = originValue & 0xff;
> highByte -= Byte.MIN_VALUE;
> return (byte)highByte;
> }
> private byte convertSimplified(byte originValue) {
> return (byte) (originValue - Byte.MIN_VALUE); // no need to byte and 0xFF.
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol merged pull request #20443: [hotfix] [FLINK-28341] [streaming-java] Change the MIN and MAX value for Byte data type
zentol merged PR #20443: URL: https://github.com/apache/flink/pull/20443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #20524: apache-flink-libraries version update
zentol closed pull request #20524: apache-flink-libraries version update URL: https://github.com/apache/flink/pull/20524 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29398) Utilize Rack Awareness in Flink Consumer
[ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632428#comment-17632428 ] Jeremy DeGroot commented on FLINK-29398: Someone on the dev list pointed out I had the wrong KIP referenced here. I fixed it. > Utilize Rack Awareness in Flink Consumer > > > Key: FLINK-29398 > URL: https://issues.apache.org/jira/browse/FLINK-29398 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Jeremy DeGroot >Assignee: Jeremy DeGroot >Priority: Major > > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > was implemented some time ago in Kafka. This allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > Flink Kafka consumers currently cannot easily use rack awareness if they're > deployed across multiple racks or availability zones, because they have no > control over which rack the Task Manager they'll be assigned to may be in. > This improvement proposes that a Kafka Consumer could be configured with a > callback or Future that could be run when it's being configured on the task > manager, that will set the appropriate value at runtime if a value is > provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #20273: [hotfix][annotation] Refine java doc for annotation VisibleForTesting
zentol closed pull request #20273: [hotfix][annotation] Refine java doc for annotation VisibleForTesting URL: https://github.com/apache/flink/pull/20273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29398) Utilize Rack Awareness in Flink Consumer
[ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy DeGroot updated FLINK-29398: --- Description: [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers. Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. was: [KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers. Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. > Utilize Rack Awareness in Flink Consumer > > > Key: FLINK-29398 > URL: https://issues.apache.org/jira/browse/FLINK-29398 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Jeremy DeGroot >Assignee: Jeremy DeGroot >Priority: Major > > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > was implemented some time ago in Kafka. This allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > Flink Kafka consumers currently cannot easily use rack awareness if they're > deployed across multiple racks or availability zones, because they have no > control over which rack the Task Manager they'll be assigned to may be in. > This improvement proposes that a Kafka Consumer could be configured with a > callback or Future that could be run when it's being configured on the task > manager, that will set the appropriate value at runtime if a value is > provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #20074: use-relbulider-to-build-project
zentol closed pull request #20074: use-relbulider-to-build-project URL: https://github.com/apache/flink/pull/20074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24477) Add MongoDB sink
[ https://issues.apache.org/jira/browse/FLINK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-24477. Resolution: Duplicate > Add MongoDB sink > > > Key: FLINK-24477 > URL: https://issues.apache.org/jira/browse/FLINK-24477 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Nir Tsruya >Assignee: Nir Tsruya >Priority: Minor > Labels: pull-request-available, stale-assigned > > h2. Motivation > *User stories:* > As a Flink user, I’d like to use MongoDB as sink for my data pipeline. > *Scope:* > * Implement an asynchronous sink for MongoDB inheriting the AsyncSinkBase > class. The implementation can for now reside in its own module in > flink-connectors. > * Implement an asynchornous sink writer for MongoDB by extending the > AsyncSinkWriter. The implemented Sink Writer will be used by the Sink class > that will be created as part of this story. > * Java / code-level docs. > * End to end testing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #19726: [FLINK-24477][flink-connectors] - Add mongodb sink
zentol closed pull request #19726: [FLINK-24477][flink-connectors] - Add mongodb sink URL: https://github.com/apache/flink/pull/19726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24558) dataStream can not use multiple classloaders
[
https://issues.apache.org/jira/browse/FLINK-24558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-24558.
Resolution: Won't Fix
> dataStream can not use multiple classloaders
> -
>
> Key: FLINK-24558
> URL: https://issues.apache.org/jira/browse/FLINK-24558
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Reporter: bai sui
>Assignee: bai sui
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Attachments: Flink ClassLoader优化 (1).png
>
>
> create a dataStream demo as below，in the demo，create a very simple example，
> read stream data from sourceFunction，and send it to sinkFunction without any
> processing.
> The point is，by creating the instance of SourceFunction and SinkFunction has
> used two separately URLClassLoader with different dependencies,for avoiding
> the code conflict .
> but the problem is flink client send to server ,the server side throw an
> classNotFoundException which defined the de classloader dependencies,
> Obviously the server side has not use the classloader as client side.
> how can I solve the problem ，is there any one can give me some advice ？
> thanks a lot
>
> {code:java}
> public class FlinkStreamDemo {
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SourceFunction sourceFunc = createSourceFunction();
> DataStreamSource dtoDataStreamSource = env.addSource(sourceFunc);
> SinkFunction sinkFunction = createSink();
> dtoDataStreamSource.addSink(sinkFunction);
> env.execute("flink-example");
> }
> private static SinkFunction createSink() {
> URL[] urls = new URL[]{...};
> ClassLoader classLoader = new URLClassLoader(urls);
> ServiceLoader loaders =
> ServiceLoader.load(ISinkFunctionFactory.class, classLoader);
> Iterator it = loaders.iterator();
> if (it.hasNext()) {
> return it.next().create();
> }
> throw new IllegalStateException();
> }
> private static SourceFunction createSourceFunction() {
> URL[] urls = new URL[]{...};
> ClassLoader classLoader = new URLClassLoader(urls);
> ServiceLoader loaders =
> ServiceLoader.load(ISourceFunctionFactory.class, classLoader);
> Iterator it = loaders.iterator();
> if (it.hasNext()) {
> return it.next().create();
> }
> throw new IllegalStateException();
> }
> public interface ISinkFunctionFactory {
> SinkFunction create();
> }
> public interface ISourceFunctionFactory {
> SourceFunction create();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on pull request #19726: [FLINK-24477][flink-connectors] - Add mongodb sink
zentol commented on PR #19726: URL: https://github.com/apache/flink/pull/19726#issuecomment-1311849382 Subsumed by https://github.com/apache/flink-connector-mongodb/pull/1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…
zentol closed pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c… URL: https://github.com/apache/flink/pull/17521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…
zentol commented on PR #17521: URL: https://github.com/apache/flink/pull/17521#issuecomment-1311846978 Will not be supported, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #19588: [FLINK-XXXX] [core] Add Generator source
zentol closed pull request #19588: [FLINK-] [core] Add Generator source URL: https://github.com/apache/flink/pull/19588 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #19618: [hotfix][cep][docs] Corrected version mentioned from Flink-1.3 to Flink-1.13
zentol closed pull request #19618: [hotfix][cep][docs] Corrected version mentioned from Flink-1.3 to Flink-1.13 URL: https://github.com/apache/flink/pull/19618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #19618: [hotfix][cep][docs] Corrected version mentioned from Flink-1.3 to Flink-1.13
zentol commented on PR #19618: URL: https://github.com/apache/flink/pull/19618#issuecomment-1311842511 1.3 is correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28427) MiniClusterResource should not override rest port settings
[ https://issues.apache.org/jira/browse/FLINK-28427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-28427. Resolution: Fixed master: 74569dc137038dfdb27b4216ee79cf36bbd789c9 > MiniClusterResource should not override rest port settings > -- > > Key: FLINK-28427 > URL: https://issues.apache.org/jira/browse/FLINK-28427 > Project: Flink > Issue Type: Technical Debt > Components: Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > For development purposes it is useful to be able to start the > MiniClusterResource with a fixed port. This would allow you to run a single > test with the UI running on a well-known port (8081, or whatever the user has > configured). > We can't expect users to dig through the log files to figure out what port is > used, nor should encourage them to directly use a MiniCluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #19278: [hotfix][docs] Fix naming to uppercase
zentol closed pull request #19278: [hotfix][docs] Fix naming to uppercase URL: https://github.com/apache/flink/pull/19278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #21299: [FLINK-28427][tests] MiniClusterResource respect explicit rest port configurations
zentol merged PR #21299: URL: https://github.com/apache/flink/pull/21299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #18917: error
zentol closed pull request #18917: error URL: https://github.com/apache/flink/pull/18917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
zentol closed pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies URL: https://github.com/apache/flink/pull/12929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #12929: [FLINK-18631] [serialization] Serializer for scala sealed trait hierarchies
zentol commented on PR #12929: URL: https://github.com/apache/flink/pull/12929#issuecomment-1311828123 Scala APIs are deprecated. Any potential serializer implementation for Scala types must be implemented in Java (if at all). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25875) "StatsDReporter. FilterCharacters" for special processing of the characters are comprehensive enough
[ https://issues.apache.org/jira/browse/FLINK-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-25875. Resolution: Won't Fix > "StatsDReporter. FilterCharacters" for special processing of the characters > are comprehensive enough > > > Key: FLINK-25875 > URL: https://issues.apache.org/jira/browse/FLINK-25875 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.14.3 >Reporter: 赵富午 >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Attachments: image-2022-01-29-11-55-20-400.png > > > I based on the 「org.Apache.Flink.Metrics.Statsd.StatsDReporter」 metrics > collection, query and display and use 「Graphite」, I found some flink metrics > cannot be queried to, after screening, I found the reason, These indicators > cannot be parsed because they contain space characters. I further track > source code, I found 「StatsDReporter.FilterCharacters」 function for the > processing of special characters is not qualified, only to replace ":" > character, for other special characters and didn't do a good replacement, > such as a space character. > !image-2022-01-29-11-55-38-064.png! > [https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#garbagecollection] > JVM "Garbagecollection" grouping, indicators have space characters, > indicators cannot be used because indicators cannot be correctly parsed and > stored in the database. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #18557: https://issues.apache.org/jira/browse/FLINK-25875
zentol closed pull request #18557: https://issues.apache.org/jira/browse/FLINK-25875 URL: https://github.com/apache/flink/pull/18557 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #18275: [FLINK-11838] please ignore this PR
zentol closed pull request #18275: [FLINK-11838] please ignore this PR URL: https://github.com/apache/flink/pull/18275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #17910: [FLINK-25056][Web Frontend] Modify Flink dashboard task manager page,…
zentol closed pull request #17910: [FLINK-25056][Web Frontend] Modify Flink dashboard task manager page,… URL: https://github.com/apache/flink/pull/17910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24731) Add a blank space for TransitiveClosureNaive
[
https://issues.apache.org/jira/browse/FLINK-24731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-24731.
Resolution: Won't Fix
> Add a blank space for TransitiveClosureNaive
>
>
> Key: FLINK-24731
> URL: https://issues.apache.org/jira/browse/FLINK-24731
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Affects Versions: 1.13.3
>Reporter: liyunchao
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> pull-request-available
>
> In
> flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/graph/TransitiveClosureNaive.scala
> Before:
> {code:java}
> (left, right) => (left._1,right._2)
> {code}
> After:
> {code:java}
> (left, right) => (left._1, right._2)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] ericxiao251 commented on pull request #21077: [FLINK-29498] Add Scala Async Retry Strategies and ResultPredicates Helper Classes
ericxiao251 commented on PR #21077: URL: https://github.com/apache/flink/pull/21077#issuecomment-1311823050 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21294: [FLINK-29704][runtime][security] E2E test for delegation token framework
gaborgsomogyi commented on PR #21294: URL: https://github.com/apache/flink/pull/21294#issuecomment-1311822976 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #17639: [FLINK-24731] add a blank space
zentol closed pull request #17639: [FLINK-24731] add a blank space URL: https://github.com/apache/flink/pull/17639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #17365: [hotfix][web]Modify the configuration of prettier
zentol commented on PR #17365: URL: https://github.com/apache/flink/pull/17365#issuecomment-1311822040 Already fixed on master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #17365: [hotfix][web]Modify the configuration of prettier
zentol closed pull request #17365: [hotfix][web]Modify the configuration of prettier URL: https://github.com/apache/flink/pull/17365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #17383: [hotfix][build] Add java8 profile
zentol closed pull request #17383: [hotfix][build] Add java8 profile URL: https://github.com/apache/flink/pull/17383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #16507: [hotfix][docs] Format repair
zentol closed pull request #16507: [hotfix][docs] Format repair URL: https://github.com/apache/flink/pull/16507 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #16507: [hotfix][docs] Format repair
zentol commented on PR #16507: URL: https://github.com/apache/flink/pull/16507#issuecomment-1311817577 1.13 is no longer maintained. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #16495: [hotfix][docs][deployment]: update broken download url
zentol closed pull request #16495: [hotfix][docs][deployment]: update broken download url URL: https://github.com/apache/flink/pull/16495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #16495: [hotfix][docs][deployment]: update broken download url
zentol commented on PR #16495: URL: https://github.com/apache/flink/pull/16495#issuecomment-1311816613 No longer applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #16189: [hotfix][docs] libs cep Remove redundant sentences
zentol merged PR #16189: URL: https://github.com/apache/flink/pull/16189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #16092: [docs][hotfix] Fix wrong scala syntax in a code snippet
zentol merged PR #16092: URL: https://github.com/apache/flink/pull/16092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29985) TaskManager might not close SlotTable on SIGTERM
[ https://issues.apache.org/jira/browse/FLINK-29985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-29985: -- Priority: Minor (was: Major) > TaskManager might not close SlotTable on SIGTERM > > > Key: FLINK-29985 > URL: https://issues.apache.org/jira/browse/FLINK-29985 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.16.0, 1.15.3 >Reporter: Roman Khachatryan >Priority: Minor > > When TM is stopped by RM, its slot table is closed, causing all its slots to > be released. > However, when TM is stopped by SIGTERM (i.e. external resource manager), its > slot table is NOT closed. > > When a slot is released, the associated resources are released as well, in > particular, MemoryManager. > MemoryManager might hold not only memory, but also arbitrary shared resources > (currently, PythonSharedResources and RocksDBSharedResources). > As of now, RocksDBSharedResources contains only ephemeral resources. Not sure > about PythonSharedResources, but likely it is associated with a separate > process. > That means that in standalone clusters, some resources might not be released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29985) TaskManager doesn't close SlotTable on SIGTERM
[ https://issues.apache.org/jira/browse/FLINK-29985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632408#comment-17632408 ] Roman Khachatryan commented on FLINK-29985: --- I found that TM actually does try to release the slot table, but there are two hard-coded timeouts: 1. 5s JvmShutdownSafeguard 2. 10s in flink-daemon.sh [cluster.services.shutdown-timeout|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#cluster-services-shutdown-timeout] is not taken into account by TM. Another issue is that even if the above timeouts don't fire, the logging system is stopped prematurely. That's why SlotTable release can be is silent. > TaskManager doesn't close SlotTable on SIGTERM > -- > > Key: FLINK-29985 > URL: https://issues.apache.org/jira/browse/FLINK-29985 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.16.0, 1.15.3 >Reporter: Roman Khachatryan >Priority: Major > > When TM is stopped by RM, its slot table is closed, causing all its slots to > be released. > However, when TM is stopped by SIGTERM (i.e. external resource manager), its > slot table is NOT closed. > > When a slot is released, the associated resources are released as well, in > particular, MemoryManager. > MemoryManager might hold not only memory, but also arbitrary shared resources > (currently, PythonSharedResources and RocksDBSharedResources). > As of now, RocksDBSharedResources contains only ephemeral resources. Not sure > about PythonSharedResources, but likely it is associated with a separate > process. > That means that in standalone clusters, some resources might not be released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29985) TaskManager might not close SlotTable on SIGTERM
[ https://issues.apache.org/jira/browse/FLINK-29985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-29985: -- Summary: TaskManager might not close SlotTable on SIGTERM (was: TaskManager doesn't close SlotTable on SIGTERM) > TaskManager might not close SlotTable on SIGTERM > > > Key: FLINK-29985 > URL: https://issues.apache.org/jira/browse/FLINK-29985 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.16.0, 1.15.3 >Reporter: Roman Khachatryan >Priority: Major > > When TM is stopped by RM, its slot table is closed, causing all its slots to > be released. > However, when TM is stopped by SIGTERM (i.e. external resource manager), its > slot table is NOT closed. > > When a slot is released, the associated resources are released as well, in > particular, MemoryManager. > MemoryManager might hold not only memory, but also arbitrary shared resources > (currently, PythonSharedResources and RocksDBSharedResources). > As of now, RocksDBSharedResources contains only ephemeral resources. Not sure > about PythonSharedResources, but likely it is associated with a separate > process. > That means that in standalone clusters, some resources might not be released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #14596: [junit5-poc][tests] Use vintage runner instead of JUnit4.
zentol closed pull request #14596: [junit5-poc][tests] Use vintage runner instead of JUnit4. URL: https://github.com/apache/flink/pull/14596 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #14695: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process
zentol closed pull request #14695: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process URL: https://github.com/apache/flink/pull/14695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #14695: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process
zentol commented on PR #14695: URL: https://github.com/apache/flink/pull/14695#issuecomment-1311809984 https://github.com/apache/flink-connector-elasticsearch/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #14696: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process
zentol closed pull request #14696: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process URL: https://github.com/apache/flink/pull/14696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #14696: [FLINK-21022] [flink-connector-es] add onSuccess handler after bulk process
zentol commented on PR #14696: URL: https://github.com/apache/flink/pull/14696#issuecomment-1311809896 Elasticsearch connector is now maintained at https://github.com/apache/flink-connector-elasticsearch/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #13693: [Hotfix][Connector-hive] Fix the banned dependency of "_2.10" for sca…
zentol closed pull request #13693: [Hotfix][Connector-hive] Fix the banned dependency of "_2.10" for sca… URL: https://github.com/apache/flink/pull/13693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #13693: [Hotfix][Connector-hive] Fix the banned dependency of "_2.10" for sca…
zentol commented on PR #13693: URL: https://github.com/apache/flink/pull/13693#issuecomment-1311808112 No longer applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #13082: added code of conduct to project
zentol closed pull request #13082: added code of conduct to project URL: https://github.com/apache/flink/pull/13082 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17445) Allow OperatorTransformation to bootstrapWith a Scala DataSet
[
https://issues.apache.org/jira/browse/FLINK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-17445.
Resolution: Won't Fix
Scala APIs are deprecated and will not be further extended.
> Allow OperatorTransformation to bootstrapWith a Scala DataSet
> -
>
> Key: FLINK-17445
> URL: https://issues.apache.org/jira/browse/FLINK-17445
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Brandon Bevans
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available
>
> Currently there is no way to use OperatorTransformation.bootstrapWith using a
> Scala DataSet. The functionality does seem to be there since a Scala DataSet
> appears to just be a wrapped Java DataSet. This functionality could be
> achieved with minimal invasiveness by passing through the underlying Java
> DataSet to the call.
>
> Also, the docs suggest that this function should work:
> From
> [https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html]:
> {code:java}
> val currencyDataSet = bEnv.fromCollection(CurrencyRate("USD", 1.0),
> CurrencyRate("EUR", 1.3)) val broadcastTransformation =
> OperatorTransformation .bootstrapWith(currencyDataSet) .transform(new
> CurrencyBootstrapFunction){code}
> I would love to take this issue on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #12285: [FLINK-17445][State Processor] Add Scala support for OperatorTransformation
zentol closed pull request #12285: [FLINK-17445][State Processor] Add Scala support for OperatorTransformation URL: https://github.com/apache/flink/pull/12285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #12285: [FLINK-17445][State Processor] Add Scala support for OperatorTransformation
zentol commented on PR #12285: URL: https://github.com/apache/flink/pull/12285#issuecomment-1311806511 Scala APIs are deprecated and will not be further extended. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #9956: [FLINK-13848][runtime] Enriches RpcEndpoint.MainThreadExecutor by supporting periodic scheduling
zentol closed pull request #9956: [FLINK-13848][runtime] Enriches RpcEndpoint.MainThreadExecutor by supporting periodic scheduling URL: https://github.com/apache/flink/pull/9956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #14595: [FLINK-20871][API/DataStream] Make DataStream#executeAndCollectWithCl…
zentol closed pull request #14595: [FLINK-20871][API/DataStream] Make DataStream#executeAndCollectWithCl… URL: https://github.com/apache/flink/pull/14595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #14595: [FLINK-20871][API/DataStream] Make DataStream#executeAndCollectWithCl…
zentol commented on PR #14595: URL: https://github.com/apache/flink/pull/14595#issuecomment-1311803466 Use `DataStream#collectAsync` + `StreamExEnv#executeAsync`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-11643) Remove useless code of externalizedCheckpointsDir
[
https://issues.apache.org/jira/browse/FLINK-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-11643.
Resolution: Fixed
> Remove useless code of externalizedCheckpointsDir
> -
>
> Key: FLINK-11643
> URL: https://issues.apache.org/jira/browse/FLINK-11643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Reporter: yuqi
>Assignee: yuqi
>Priority: Not a Priority
> Labels: pull-request-available, stale-assigned
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In ExecutionGraphBuilder#buildGraph, there exists unless code as follows:
> {code:java}
> // The default directory for externalized checkpoints
> String externalizedCheckpointsDir =
> jobManagerConfig.getString(CheckpointingOptions.CHECKPOINTS_DIRECTORY);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #8150: [FLINK-12162][Table SQL]Fix build error in flink-table-planner
zentol closed pull request #8150: [FLINK-12162][Table SQL]Fix build error in flink-table-planner URL: https://github.com/apache/flink/pull/8150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #7729: [FLINK-11643] remove useless code of externalizedCheckpointsDir
zentol closed pull request #7729: [FLINK-11643] remove useless code of externalizedCheckpointsDir URL: https://github.com/apache/flink/pull/7729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #7729: [FLINK-11643] remove useless code of externalizedCheckpointsDir
zentol commented on PR #7729: URL: https://github.com/apache/flink/pull/7729#issuecomment-1311800937 No longer applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #15654: [FLINK-22141] Bumps checkstyle, junit, log4j version for security
zentol closed pull request #15654: [FLINK-22141] Bumps checkstyle, junit, log4j version for security URL: https://github.com/apache/flink/pull/15654 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #15654: [FLINK-22141] Bumps checkstyle, junit, log4j version for security
zentol commented on PR #15654: URL: https://github.com/apache/flink/pull/15654#issuecomment-1311799865 Log4j/junit were upgraded in the meantime. Checkstyle upgrade is tracked in FLINK-23542. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-19159. Resolution: Duplicate > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement > Components: API / Scala >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #8427: [hotfix][docs]Correct the class name of JSONDeserializationSchema in kafka connector desc
zentol closed pull request #8427: [hotfix][docs]Correct the class name of JSONDeserializationSchema in kafka connector desc URL: https://github.com/apache/flink/pull/8427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #8427: [hotfix][docs]Correct the class name of JSONDeserializationSchema in kafka connector desc
zentol commented on PR #8427: URL: https://github.com/apache/flink/pull/8427#issuecomment-1311794331 no longer applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #7446: Added Model serving - implementation of the Flip -23.
zentol closed pull request #7446: Added Model serving - implementation of the Flip -23. URL: https://github.com/apache/flink/pull/7446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #7446: Added Model serving - implementation of the Flip -23.
zentol commented on PR #7446: URL: https://github.com/apache/flink/pull/7446#issuecomment-1311793042 Flink ML has been moved to a separate repository. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-11344) Display All Execution Attempt Information on Flink Web Dashboard
[ https://issues.apache.org/jira/browse/FLINK-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-11344. Resolution: Fixed Subsumed by FLINK-28588 > Display All Execution Attempt Information on Flink Web Dashboard > > > Key: FLINK-11344 > URL: https://issues.apache.org/jira/browse/FLINK-11344 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: BoWang >Assignee: BoWang >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, only one Execution Attempt of each sub-task is shown in web > dashboard, thus, only the succeed Attempt is shown when failover occurs. This > may be inconvenient to rapidly locate the failure reasons of failed Attempts -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol closed pull request #7560: [FLINK-11344][webfrontend] display all execution attempts on web
zentol closed pull request #7560: [FLINK-11344][webfrontend] display all execution attempts on web URL: https://github.com/apache/flink/pull/7560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #7560: [FLINK-11344][webfrontend] display all execution attempts on web
zentol commented on PR #7560: URL: https://github.com/apache/flink/pull/7560#issuecomment-1311791930 Subsumed by FLINK-28588. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-9609) Add bucket ready mechanism for BucketingSink when checkpoint complete
[
https://issues.apache.org/jira/browse/FLINK-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-9609.
---
Resolution: Won't Fix
The BucketingSink has been removed.
> Add bucket ready mechanism for BucketingSink when checkpoint complete
> -
>
> Key: FLINK-9609
> URL: https://issues.apache.org/jira/browse/FLINK-9609
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common, Connectors / FileSystem
>Affects Versions: 1.4.2, 1.5.0
>Reporter: zhangminglei
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available
>
> Currently, BucketingSink only support {{notifyCheckpointComplete}}. However,
> users want to do some extra work when a bucket is ready. It would be nice if
> we can support {{BucketReady}} mechanism for users or we can tell users when
> a bucket is ready for use. For example, One bucket is created for every 5
> minutes, at the end of 5 minutes before creating the next bucket, the user
> might need to do something as the previous bucket ready, like sending the
> timestamp of the bucket ready time to a server or do some other stuff.
> Here, Bucket ready means all the part files suffix name under a bucket
> neither {{.pending}} nor {{.in-progress}}. Then we can think this bucket is
> ready for user use. Like a watermark means no elements with a timestamp older
> or equal to the watermark timestamp should arrive at the window. We can also
> refer to the concept of watermark here, or we can call this *BucketWatermark*
> if we could.
> Recently, I found a user who wants this functionality which I would think.
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Let-BucketingSink-roll-file-on-each-checkpoint-td19034.html
> Below is what he said:
> My user case is we read data from message queue, write to HDFS, and our ETL
> team will use the data in HDFS. *In the case, ETL need to know if all data is
> ready to be read accurately*, so we use a counter to count how many data has
> been wrote, if the counter is equal to the number we received, we think HDFS
> file is ready. We send the counter message in a custom sink so ETL can know
> how many data has been wrote, but if use current BucketingSink, even through
> HDFS file is flushed, ETL may still cannot read the data. If we can close
> file during checkpoint, then the result is accurately. And for the HDFS small
> file problem, it can be controller by use bigger checkpoint interval.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on pull request #6375: [FLINK-9609] [connectors] Add bucket ready mechanism for BucketingSin…
zentol commented on PR #6375: URL: https://github.com/apache/flink/pull/6375#issuecomment-1311789090 The BucketingSink has been removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #6375: [FLINK-9609] [connectors] Add bucket ready mechanism for BucketingSin…
zentol closed pull request #6375: [FLINK-9609] [connectors] Add bucket ready mechanism for BucketingSin… URL: https://github.com/apache/flink/pull/6375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1
zentol closed pull request #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1 URL: https://github.com/apache/flink/pull/6365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #6365: [FLINK-9849] [hbase] Hbase upgrade to 2.0.1
zentol commented on PR #6365: URL: https://github.com/apache/flink/pull/6365#issuecomment-1311788087 Subsumed by FLINK-18795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol closed pull request #7865: [FLINK-9650] [formats] add support for protobuf objects
zentol closed pull request #7865: [FLINK-9650] [formats] add support for protobuf objects URL: https://github.com/apache/flink/pull/7865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #7865: [FLINK-9650] [formats] add support for protobuf objects
zentol commented on PR #7865: URL: https://github.com/apache/flink/pull/7865#issuecomment-1311785611 Subsumed by FLINK-18202. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29990) Unparsed SQL for SqlTableLike cannot be parsed correctly
[
https://issues.apache.org/jira/browse/FLINK-29990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benchao Li closed FLINK-29990.
--
Assignee: Shuo Cheng
Resolution: Fixed
Fixed via
[https://github.com/apache/flink/commit/dd76844342489a252f8b76417090f137028af0bc]
[~icshuo] thanks for reporting and fixing this.
> Unparsed SQL for SqlTableLike cannot be parsed correctly
>
>
> Key: FLINK-29990
> URL: https://issues.apache.org/jira/browse/FLINK-29990
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Shuo Cheng
>Assignee: Shuo Cheng
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> Consider the following DDL sql (LIKE without any options):
> {code:java}
> create table source_table(
> a int,
> b bigint,
> c string
> )
> LIKE parent_table{code}
> After unparsed by sql parser, we get the following result:
> {code:java}
> CREATE TABLE `SOURCE_TABLE` (
> `A` INTEGER,
> `B` BIGINT,
> `C` STRING
> )
> LIKE `PARENT_TABLE` (
> ) {code}
> Exception will be thrown if you try to parse the above sql.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] libenchao merged pull request #21291: [FLINK-29990][sql-parser] Fix unparsed sql for SqlTableLike with no o…
libenchao merged PR #21291: URL: https://github.com/apache/flink/pull/21291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29609) Clean up jobmanager deployment on suspend after recording savepoint info
[ https://issues.apache.org/jira/browse/FLINK-29609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29609: --- Labels: pull-request-available (was: ) > Clean up jobmanager deployment on suspend after recording savepoint info > > > Key: FLINK-29609 > URL: https://issues.apache.org/jira/browse/FLINK-29609 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.3.0 > > > Currently in case of suspending with savepoint. The jobmanager pod will > linger there forever after cancelling the job. > This is currently used to ensure consistency in case the > operator/cancel-with-savepoint operation fails. > Once we are sure however that the savepoint has been recorded and the job is > shut down, we should clean up all the resources. Optionally we can make this > configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
