[jira] [Updated] (FLINK-553) Add getGroupKey() method to group-at-time operators
[ https://issues.apache.org/jira/browse/FLINK-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-553: - Labels: auto-deprioritized-major github-import stale-minor (was: auto-deprioritized-major github-import) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add getGroupKey() method to group-at-time operators > --- > > Key: FLINK-553 > URL: https://issues.apache.org/jira/browse/FLINK-553 > Project: Flink > Issue Type: Improvement > Components: API / DataSet >Reporter: GitHub Import >Priority: Minor > Labels: auto-deprioritized-major, github-import, stale-minor > > Group-at-a-time operators (Reduce & CoGroup) work on multiple records in one > UDF call. Often these UDFs need to access the key that is common to all > records of a group. > We could add a function to set a the key of a group before the UDF is called > (``setGroupKey()``) and a function to get the key (``getGroupKey()``) that > can be called from the UDF. > What do you think about this? > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/553 > Created by: [fhueske|https://github.com/fhueske] > Labels: enhancement, java api, scala api, user satisfaction, > Assignee: [aalexandrov|https://github.com/aalexandrov] > Created at: Mon Mar 10 22:28:27 CET 2014 > State: open -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-951) Reworking of Iteration Synchronization, Accumulators and Aggregators
[ https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-951: - Labels: auto-deprioritized-major auto-unassigned refactoring stale-minor (was: auto-deprioritized-major auto-unassigned refactoring) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Reworking of Iteration Synchronization, Accumulators and Aggregators > > > Key: FLINK-951 > URL: https://issues.apache.org/jira/browse/FLINK-951 > Project: Flink > Issue Type: Improvement > Components: API / DataSet, Runtime / Task >Affects Versions: 0.9 >Reporter: Markus Holzemer >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, refactoring, > stale-minor > Original Estimate: 168h > Remaining Estimate: 168h > > I just realized that there is no real Jira issue for the task I am currently > working on. > I am currently reworking a few things regarding Iteration Synchronization, > Accumulators and Aggregators. Currently the synchronization at the end of one > superstep is done through channel events. That makes it hard to track the > current status of iterations. That is why I am changing this synchronization > to use RPC calls with the JobManager, so that the JobManager manages the > current status of all iterations. > Currently we use Accumulators outside of iterations and Aggregators inside of > iterations. Both have a similiar function, but a bit different interfaces and > handling. I want to unify these two concepts. I propose that we stick in the > future to Accumulators only. Aggregators therefore are removed and > Accumulators are extended to cover the usecases Aggregators were used fore > before. The switch to RPC for iterations makes it possible to also send the > current Accumulator values at the end of each superstep, so that the > JobManager (and thereby the webinterface) will be able to print intermediate > accumulation results. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-1526) Add Minimum Spanning Tree library method and example
[ https://issues.apache.org/jira/browse/FLINK-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1526: -- Labels: algorithm auto-deprioritized-major auto-unassigned stale-minor (was: algorithm auto-deprioritized-major auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add Minimum Spanning Tree library method and example > > > Key: FLINK-1526 > URL: https://issues.apache.org/jira/browse/FLINK-1526 > Project: Flink > Issue Type: New Feature > Components: Library / Graph Processing (Gelly) >Reporter: Vasia Kalavri >Priority: Minor > Labels: algorithm, auto-deprioritized-major, auto-unassigned, > stale-minor > > This issue proposes the addition of a library method and an example for > distributed minimum spanning tree in Gelly. > The DMST algorithm is very interesting because it is quite different from > PageRank-like iterative graph algorithms. It consists of distinct phases > inside the same iteration and requires a mechanism to detect convergence of > one phase to proceed to the next one. Current implementations in > vertex-centric models are quite long (>1000 lines) and hard to understand. > You can find a description of the algorithm [here | > http://ilpubs.stanford.edu:8090/1077/3/p535-salihoglu.pdf] and [here | > http://www.vldb.org/pvldb/vol7/p1047-han.pdf]. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-1904) Add cross() to Table API
[ https://issues.apache.org/jira/browse/FLINK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1904: -- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add cross() to Table API > > > Key: FLINK-1904 > URL: https://issues.apache.org/jira/browse/FLINK-1904 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Aljoscha Krettek >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-1493) Support for streaming jobs preserving global ordering of records
[ https://issues.apache.org/jira/browse/FLINK-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1493: -- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support for streaming jobs preserving global ordering of records > > > Key: FLINK-1493 > URL: https://issues.apache.org/jira/browse/FLINK-1493 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Márton Balassi >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > Distributed streaming jobs do not give total, global ordering guarantees for > records only partial ordering is provided by the system: records travelling > on the same exact route of the physical plan are ordered, but they aren't > between routes. > It turns out that although this feature can only be implemented via "merge > sorting" in the input buffers on a timestamp field thus creating substantial > latency is still desired for a number of applications. > Just a heads up for the implementation: the sorting introduces back pressure > in the buffers and might cause deadlocks. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-8003) Support named ROW in Flink SQL
[
https://issues.apache.org/jira/browse/FLINK-8003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8003:
--
Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
(was: auto-deprioritized-major auto-unassigned stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Support named ROW in Flink SQL
> --
>
> Key: FLINK-8003
> URL: https://issues.apache.org/jira/browse/FLINK-8003
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Shuyi Chen
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> We want to use the row value constructor and CAST to create a named row, or
> even nested named row, e.g.
> {code:sql}
> CREATE TYPE myrowtype AS (f1 INTEGER, f2 VARCHAR(10))
> SELECT CAST(ROW(intField, "test") AS myrowtype) AS myRow FROM myTable;
> {code}
> So if converted to JSON, the output will be
> {code:java}
> {"myRow":{"f0": ${intField}, "f1":"test"}}
> {code}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-1814) Revisit the documentation to add new operator
[ https://issues.apache.org/jira/browse/FLINK-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1814: -- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Revisit the documentation to add new operator > - > > Key: FLINK-1814 > URL: https://issues.apache.org/jira/browse/FLINK-1814 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Henry Saputra >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > The doc to describe adding new operator seem to have broken links and may not > be accurate anymore [1] > This ticket is filed to revisit the doc and update it if necessary > [1] > https://ci.apache.org/projects/flink/flink-docs-master/internals/add_operator.html -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-1689) Add documentation on streaming file sinks interaction with the batch outputformat
[ https://issues.apache.org/jira/browse/FLINK-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1689: -- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add documentation on streaming file sinks interaction with the batch > outputformat > - > > Key: FLINK-1689 > URL: https://issues.apache.org/jira/browse/FLINK-1689 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Márton Balassi >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > OutputFormats supported by the batch API are supported in streaming through > the FileSinkFunction. A bit of documentation on that is needed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-987) Extend TypeSerializers and -Comparators to work directly on Memory Segments
[
https://issues.apache.org/jira/browse/FLINK-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-987:
-
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Extend TypeSerializers and -Comparators to work directly on Memory Segments
> ---
>
> Key: FLINK-987
> URL: https://issues.apache.org/jira/browse/FLINK-987
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 0.6-incubating
>Reporter: Stephan Ewen
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> As per discussion with [~till.rohrmann], [~uce], [~aljoscha], we suggest to
> change the way that the TypeSerialzers/Comparators and
> DataInputViews/DataOutputViews work.
> The goal is to allow more flexibility in the construction on the binary
> representation of data types, and to allow partial deserialization of
> individual fields. Both is currently prohibited by the fact that the
> abstraction of the memory (into which the data goes) is a stream abstraction
> ({{DataInputView}}, {{DataOutputView}}).
> An idea is to offer a random-access buffer like view for construction and
> random-access deserialization, as well as various methods to copy elements in
> a binary fashion between such buffers and streams.
> A possible set of methods for the {{TypeSerializer}} could be:
> {code}
> long serialize(T record, TargetBuffer buffer);
>
> T deserialize(T reuse, SourceBuffer source);
>
> void ensureBufferSufficientlyFilled(SourceBuffer source);
>
> X deserializeField(X reuse, int logicalPos, SourceBuffer buffer);
>
> int getOffsetForField(int logicalPos, int offset, SourceBuffer buffer);
>
> void copy(DataInputView in, TargetBuffer buffer);
>
> void copy(SourceBuffer buffer,, DataOutputView out);
>
> void copy(DataInputView source, DataOutputView target);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-7607) Web Frontend Hangs with Large Numbers of Tasks
[ https://issues.apache.org/jira/browse/FLINK-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-7607: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned performance (was: auto-deprioritized-major auto-unassigned performance stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Web Frontend Hangs with Large Numbers of Tasks > -- > > Key: FLINK-7607 > URL: https://issues.apache.org/jira/browse/FLINK-7607 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.3.2 > Environment: Attempted to load the web frontend on a MacBook Pro 15" > (late 2016) with 16 GB of memory using both Chrome 60.0 and Safari 10.1.2. >Reporter: Joshua Griffith >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned, performance > > Viewing a job with a high number of tasks in the web front-end causes the > page to hang, consuming 100% CPU on a core. At 200 tasks the page slows > noticeably and scrolling results in long, non-responsive pauses. At 400 tasks > the page only updates once per minute and is almost entirely non-responsive. > Initially, I thought this was caused by rendering a complex job graph but > opening the inspector and deleting the canvas did not improve page > performance. Further inspection indicated that the page was redrawing every > DOM element in the task list on every update. > A possible solution is to use an approach similar to > [react-list|https://github.com/orgsync/react-list] and only request > data/render list items that are in view and only update DOM nodes that have > changed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-7546) Support SUBMULTISET_OF Operator for Multiset SQL type
[ https://issues.apache.org/jira/browse/FLINK-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-7546: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned (was: auto-deprioritized-major auto-unassigned stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support SUBMULTISET_OF Operator for Multiset SQL type > - > > Key: FLINK-7546 > URL: https://issues.apache.org/jira/browse/FLINK-7546 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Shuyi Chen >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-7859) Allow field annotations to define TypeInformation
[
https://issues.apache.org/jira/browse/FLINK-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-7859:
--
Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
(was: auto-deprioritized-major auto-unassigned stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Allow field annotations to define TypeInformation
> -
>
> Key: FLINK-7859
> URL: https://issues.apache.org/jira/browse/FLINK-7859
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.3.2
>Reporter: Timo Walther
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> Right now it is impossible to define custom TypeInformation for specific
> fields. For example, we cannot provide information about a {{Row}} type that
> is a field of Scala case class. An option would be to allow fields to be
> annotated with {{@TypeInfo}} such that a type information can be provided.
> {code}
> case class Test(a: Int, b: String, @TypeInfo(classOf[MyRowFactory]) c: Row)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-8236) Allow to set the parallelism of table queries
[
https://issues.apache.org/jira/browse/FLINK-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8236:
--
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Allow to set the parallelism of table queries
> -
>
> Key: FLINK-8236
> URL: https://issues.apache.org/jira/browse/FLINK-8236
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.4.0
>Reporter: Timo Walther
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> Right now the parallelism of a table program is determined by the parallelism
> of the stream/batch environment. E.g., by default, tumbling window operators
> use the default parallelism of the environment. Simple project and select
> operations have the same parallelism as the inputs they are applied on.
> While we cannot change forwarding operations because this would change the
> results when using retractions, it should be possible to change the
> parallelism for operators after shuffling operations.
> It should be possible to specify the default parallelism of a table program
> in the {{TableConfig}} and/or {{QueryConfig}}. The configuration per query
> has higher precedence that the configuration per table environment.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-7554) Add a testing RuntimeContext to test utilities
[
https://issues.apache.org/jira/browse/FLINK-7554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-7554:
--
Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
pull-request-available starter (was: auto-deprioritized-major auto-unassigned
pull-request-available stale-minor starter)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Add a testing RuntimeContext to test utilities
> --
>
> Key: FLINK-7554
> URL: https://issues.apache.org/jira/browse/FLINK-7554
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream, Tests
>Reporter: Timo Walther
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available, starter
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unit testing user-defined functions it would be useful to have an
> official testing {{RuntimeContext}} that uses Java collections for storing
> state, metrics, etc.
> After executing the business logic, the user could then verify how the state
> of the UDF changed or which metrics have been collected.
> This issue includes documentation for the "Testing" section.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-8180) Refactor driver outputs
[ https://issues.apache.org/jira/browse/FLINK-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-8180: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned (was: auto-deprioritized-major auto-unassigned stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Refactor driver outputs > --- > > Key: FLINK-8180 > URL: https://issues.apache.org/jira/browse/FLINK-8180 > Project: Flink > Issue Type: Improvement > Components: Library / Graph Processing (Gelly) >Affects Versions: 1.5.0 >Reporter: Greg Hogan >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > > The change in 1.4 of algorithm results from Tuples to POJOs broke the writing > of results as csv. Testing this was and is a challenge so was not done. There > are many additional improvements which can be made based on recent > improvements to the Gelly framework. > Result hash and analytic results should always be printed to the screen. > Results can optionally be written to stdout or to a file. In the latter case > the result hash and analytic results (and schema) will also be written to a > top-level file. > The "verbose" output strings can be replaced with json which is just as > human-readable but also machine readable. In addition to csv and json it may > be simple to support xml, etc. Computed fields will be optionally printed to > screen or file (currently these are always printed to screen but never to > file). > Testing will be simplified since formats are now a separate concern from the > stream. > Jackson is available to Gelly as a dependency provided in the Flink > distribution but we may want to build flink-gelly-examples as an uber jar in > order to include additional modules (which may require a direct dependency on > Jackson, which would require checkstyle suppressions around the unshaded > jackson imports). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-7946) Add an option to truncate IDs to 8 characters
[ https://issues.apache.org/jira/browse/FLINK-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-7946: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned (was: auto-deprioritized-major auto-unassigned stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add an option to truncate IDs to 8 characters > - > > Key: FLINK-7946 > URL: https://issues.apache.org/jira/browse/FLINK-7946 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > > This issue is about adding a short_ids option to the metrics configuration to > truncate all IDs to 8 characters. > The string representation for Flink IDs is 32 characters long; this level of > uniqueness is however not required in all cases. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-7913) Add support for Kafka default partitioner
[ https://issues.apache.org/jira/browse/FLINK-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-7913: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned (was: auto-deprioritized-major auto-unassigned stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add support for Kafka default partitioner > - > > Key: FLINK-7913 > URL: https://issues.apache.org/jira/browse/FLINK-7913 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.4.0 >Reporter: Konstantin Lalafaryan >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > > Currently in the Apache Flink it is available only *FlinkKafkaPartitioner* > and just one implementation *FlinkFixedPartitioner*. > In order to be able to use Kafka's default partitioner you have to create new > implementation for *FlinkKafkaPartitioner* and fork the code from the Kafka. > It will be really good to be able to define the partitioner without > implementing the new class. > Thanks. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-8090) Improve error message when registering different states under the same name.
[
https://issues.apache.org/jira/browse/FLINK-8090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-8090:
--
Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
pull-request-available (was: auto-deprioritized-major auto-unassigned
pull-request-available stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Improve error message when registering different states under the same name.
>
>
> Key: FLINK-8090
> URL: https://issues.apache.org/jira/browse/FLINK-8090
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.4.0
>Reporter: Kostas Kloudas
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently a {{ProcessFunction}} like this:
> {code}
> final MapStateDescriptor>
> firstMapStateDescriptor = new MapStateDescriptor<>(
> "timon-one",
> BasicTypeInfo.INT_TYPE_INFO,
> source.getType());
> final ListStateDescriptor secondListStateDescriptor = new
> ListStateDescriptor(
> "timon-one",
> BasicTypeInfo.INT_TYPE_INFO);
> new ProcessFunction, Object>() {
> private static final long serialVersionUID =
> -805125545438296619L;
> private transient MapState Tuple2> firstMapState;
> private transient ListState
> secondListState;
> @Override
> public void open(Configuration parameters)
> throws Exception {
> super.open(parameters);
> firstMapState =
> getRuntimeContext().getMapState(firstMapStateDescriptor);
> secondListState =
> getRuntimeContext().getListState(secondListStateDescriptor);
> }
> @Override
> public void processElement(Tuple2 Long> value, Context ctx, Collector out) throws Exception {
> Tuple2 v =
> firstMapState.get(value.f0);
> if (v == null) {
> v = new Tuple2<>(value.f0, 0L);
> }
> firstMapState.put(value.f0, new
> Tuple2<>(v.f0, v.f1 + value.f1));
> }
> }
> {code}
> fails with:
> {code}
> java.lang.RuntimeException: Error while getting state
> at
> org.apache.flink.runtime.state.DefaultKeyedStateStore.getListState(DefaultKeyedStateStore.java:74)
> at
> org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getListState(StreamingRuntimeContext.java:127)
> at
> org.apache.flink.queryablestate.itcases.AbstractQueryableStateTestBase$2.open(AbstractQueryableStateTestBase.java:327)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:58)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:381)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException:
> org.apache.flink.runtime.state.heap.HeapMapState cannot be cast to
> org.apache.flink.api.common.state.ListState
> at
> org.apache.flink.runtime.state.DefaultKeyedStateStore.getListState(DefaultKeyedStateStore.java:71)
> ... 9 more
> {code}
> Which is cryptic, as it does not explain the reason for the problem. The
> error message should be something along the line of "Duplicate state name".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18251: [FLINK-25488][table] Clarify delimiters usage in STR_TO_MAP function
flinkbot edited a comment on pull request #18251: URL: https://github.com/apache/flink/pull/18251#issuecomment-1003446437 ## CI report: * db4253de50f8c4117db88f30a5d2278df4af751e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28818) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18251: [FLINK-25488][table] Clarify delimiters usage in STR_TO_MAP function
flinkbot edited a comment on pull request #18251: URL: https://github.com/apache/flink/pull/18251#issuecomment-1003446437 ## CI report: * db4253de50f8c4117db88f30a5d2278df4af751e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28818) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18251: [FLINK-25488][table] Clarify delimiters usage in STR_TO_MAP function
flinkbot commented on pull request #18251: URL: https://github.com/apache/flink/pull/18251#issuecomment-1003446600 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit db4253de50f8c4117db88f30a5d2278df4af751e (Fri Dec 31 20:24:17 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25488).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18251: [FLINK-25488][table] Clarify delimiters usage in STR_TO_MAP function
flinkbot commented on pull request #18251: URL: https://github.com/apache/flink/pull/18251#issuecomment-1003446437 ## CI report: * db4253de50f8c4117db88f30a5d2278df4af751e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25488) Using a pipe symbol as pair delimiter in STR_TO_MAP in combination with concatenation results in broken output
[
https://issues.apache.org/jira/browse/FLINK-25488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25488:
---
Labels: pull-request-available (was: )
> Using a pipe symbol as pair delimiter in STR_TO_MAP in combination with
> concatenation results in broken output
> ---
>
> Key: FLINK-25488
> URL: https://issues.apache.org/jira/browse/FLINK-25488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.14.2
>Reporter: Martijn Visser
>Priority: Major
> Labels: pull-request-available
>
> Reproducible using Flink Faker:
> {code:sql}
> -- Create source table
> CREATE TABLE `customers` (
> `identifier` STRING,
> `fullname` STRING,
> `postal_address` STRING,
> `residential_address` STRING
> ) WITH (
> 'connector' = 'faker',
> 'fields.identifier.expression' = '#{Internet.uuid}',
> 'fields.fullname.expression' = '#{Name.firstName} #{Name.lastName}',
> 'fields.postal_address.expression' = '#{Address.fullAddress}',
> 'fields.residential_address.expression' = '#{Address.fullAddress}',
> 'rows-per-second' = '1'
> );
> {code}
> {code:sql}
> -- Doesn't generate expected output
> SELECT
> `identifier`,
> `fullname`,
> STR_TO_MAP('postal_address:' || postal_address || '|residential_address:'
> || residential_address,'|',':') AS `addresses`
> FROM `customers`;
> {code}
> Output will look like:
> {code:sql}
> {=, A=null, C=null, D=null, L=null, O=null, P=null, S=null, T=null, _=null,
> =null, a=null, b=null, c=null, d=null, e=null, g=null, h=null, i=null,
> ,=null, l=null, -=null, m=null, .=null, n=null, o=null, p=null, q=null,
> 2=null, r=null, 3=null, s=null, 4=null, t=null, 5=null, u=null, 6=null,
> v=null, 7=null, w=null, 8=null, 9=null, |=null}
> {code}
> When using:
> {code:sql}
> -- Output looks like expected when using a different separator
> SELECT
> `identifier`,
> `fullname`,
> STR_TO_MAP('postal_address:' || postal_address || ';residential_address:'
> || residential_address,';',':') AS `addresses`
> FROM `customers`;
> {code}
> The output looks as expected:
> {code:sql}
> {postal_address=6654 Chong Meadows, East Lupita, CT 51702-8560,
> residential_address=Apt. 098 51845 Shields Fork, North Erikland, NV 10386}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] biyuhao opened a new pull request #18251: [FLINK-25488][table] Clarify delimiters usage in STR_TO_MAP function
biyuhao opened a new pull request #18251: URL: https://github.com/apache/flink/pull/18251 ## What is the purpose of the change To clarify delimiters usage in STR_TO_MAP function ## Brief change log - Documentation update - Trivial UT update ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing This change is already covered by existing tests, such as *ScalarFunctionsTest*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii closed pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
gaoyunhaii closed pull request #18249: URL: https://github.com/apache/flink/pull/18249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18239: [FLINK-25426][runtime] Calling super#onStop for all RpcEndpoint children to be sure that all resources close correctly
flinkbot edited a comment on pull request #18239: URL: https://github.com/apache/flink/pull/18239#issuecomment-1002763066 ## CI report: * 41470b5a91b701c601bd2966680cd52dfcc9091c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28816) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: taskmanager_thread_dump-2.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-3.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-3.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission-3.log,
> job_submission.log, pull-secret.txt, taskmanager_thread_dump-1.log,
> taskmanager_thread_dump-2.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment:
flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-3.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-3.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-3.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission-3.log,
> job_submission.log, pull-secret.txt, taskmanager_thread_dump-1.log,
> taskmanager_thread_dump-2.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: job_submission-3.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-3.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-3.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission-3.log,
> job_submission.log, pull-secret.txt, taskmanager_thread_dump-1.log,
> taskmanager_thread_dump-2.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-3.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-3.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-3.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission-3.log,
> job_submission.log, pull-secret.txt, taskmanager_thread_dump-1.log,
> taskmanager_thread_dump-2.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: aylık_toplamlar.PNG
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> pull-secret.txt, taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: Capture.PNG
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> pull-secret.txt, taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: pull-secret.txt
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> pull-secret.txt, taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment:
flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log
flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Capture.PNG,
> Screen Shot 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at
> 21.26.25.png, Screen Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> aylık_toplamlar.PNG,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-2.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-2.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> pull-secret.txt, taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: Screen Shot 2021-12-31 at 21.26.25-1.png
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Screen Shot
> 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at 21.26.25.png, Screen
> Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: Akbank Toplantı Notları.md
Akbank.drawio
Vodafone.md
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Akbank Toplantı Notları.md, Akbank.drawio, Screen Shot
> 2021-12-31 at 21.26.25-1.png, Screen Shot 2021-12-31 at 21.26.25.png, Screen
> Shot 2021-12-31 at 21.26.44.png, Vodafone.md,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment:
flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log
flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png, Screen Shot
> 2021-12-31 at 21.26.44.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25496) ThreadDumpInfoTest.testComparedWithDefaultJDKImplemetation failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467293#comment-17467293
]
Yun Gao commented on FLINK-25496:
-
Fix on master via 2db77438db3b985bb688b71f791be8e68883410f, will close issue if
it does not reproduce.
> ThreadDumpInfoTest.testComparedWithDefaultJDKImplemetation failed on azure
> ---
>
> Key: FLINK-25496
> URL: https://issues.apache.org/jira/browse/FLINK-25496
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Reporter: Yun Gao
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
>
> {code:java}
> Dec 31 02:53:26 [ERROR] Failures:
> Dec 31 02:53:26 [ERROR]
> ThreadDumpInfoTest.testComparedWithDefaultJDKImplemetation:66
> expected:<"main" [prio=5 ]Id=1 RUNNABLE
> Dec 31 02:53:26 at ja...> but was:<"main" []Id=1 RUNNABLE
> Dec 31 02:53:26 at ja...>
> Dec 31 02:53:26 [INFO]
> Dec 31 02:53:26 [ERROR] Tests run: 5958, Failures: 1, Errors: 0, Skipped: 26
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28779=logs=b0a398c0-685b-599c-eb57-c8c2a771138e=747432ad-a576-5911-1e2a-68c6bedc248a=13859
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: job_submission-2.log
taskmanager_thread_dump-1.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png, Screen Shot
> 2021-12-31 at 21.26.44.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4-1.log,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n-1.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission-2.log, job_submission.log,
> taskmanager_thread_dump-1.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: job_submission-1.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png, Screen Shot
> 2021-12-31 at 21.26.44.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission-1.log, job_submission.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] gaoyunhaii commented on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
gaoyunhaii commented on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003434470 Very thanks @zuston for the update! LGTM and manually verified the issue get fixed with JDK11. Would merge the PR with changing the comment style from `/*` to `//` so that it could take effect in the next cron job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: taskmanager_thread_dump.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png, Screen Shot
> 2021-12-31 at 21.26.44.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission.log, taskmanager_thread_dump.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: Screen Shot 2021-12-31 at 21.26.44.png
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png, Screen Shot
> 2021-12-31 at 21.26.44.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: Screen Shot 2021-12-31 at 21.26.25.png
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: job_submission.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-31 at 21.26.25.png,
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log,
> job_submission.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kerem Ulutaş updated FLINK-25023:
-
Attachment: flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log
flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
> Attachments:
> flink--standalonesession-0-flink-jobmanager-589479f45b-p66k4.log,
> flink--taskexecutor-0-flink-taskmanager-9f6685b57-vfb2n.log
>
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * b0da7ee5efc7652b302161d8eab7989a29e5160c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * 29da74c1ab8840f9a749b76f12b2aa8017d228cd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28815) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18239: [FLINK-25426][runtime] Calling super#onStop for all RpcEndpoint children to be sure that all resources close correctly
flinkbot edited a comment on pull request #18239: URL: https://github.com/apache/flink/pull/18239#issuecomment-1002763066 ## CI report: * e92561894f05a7ee7406ac0d7facd82649b583e0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28790) * 41470b5a91b701c601bd2966680cd52dfcc9091c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28816) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18239: [FLINK-25426][runtime] Calling super#onStop for all RpcEndpoint children to be sure that all resources close correctly
flinkbot edited a comment on pull request #18239: URL: https://github.com/apache/flink/pull/18239#issuecomment-1002763066 ## CI report: * e92561894f05a7ee7406ac0d7facd82649b583e0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28790) * 41470b5a91b701c601bd2966680cd52dfcc9091c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] akalash commented on pull request #18239: [FLINK-25426][runtime] Calling super#onStop for all RpcEndpoint children to be sure that all resources close correctly
akalash commented on pull request #18239: URL: https://github.com/apache/flink/pull/18239#issuecomment-1003417298 I have added one more commit with enabling SavepointITCase#testTriggerSavepointAndResumeWithNoClaim since I think it should be also fixed by this fix. Original task for this test - https://issues.apache.org/jira/browse/FLINK-25427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25427) SavepointITCase.testTriggerSavepointAndResumeWithNoClaim fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467284#comment-17467284
]
Anton Kalashnikov commented on FLINK-25427:
---
I actually think that it is the same problem as
https://issues.apache.org/jira/browse/FLINK-25426. It started to fail from one
commit. It always fails when `UnalignedCheckpointRescaleITCase` fails. When I
try to reproduce it locally I see the same exceptions as for
`UnalignedCheckpointRescaleITCase`.
Unfortunatelly, I can not be fully sure about it since the original problem
`Condition was not met in given timeout.`, doesn't contain anything in logs.
This test just stucks during allocation slot(`Allocated slot for ..`) which on
one hand points to the same problem as in
`UnalignedCheckpointRescaleITCase`(the problem with resources managment) but on
another hand I don't see explicit OOM exception there(we have problem with
logging somewhere?).
Since the fix for `UnalignedCheckpointRescaleITCase` is almost merged. If
nobody mind I can enable `SavepointITCase` there and we will see it helps or
not.
> SavepointITCase.testTriggerSavepointAndResumeWithNoClaim fails on AZP
> -
>
> Key: FLINK-25427
> URL: https://issues.apache.org/jira/browse/FLINK-25427
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Anton Kalashnikov
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.15.0
>
>
> The test {{SavepointITCase.testTriggerSavepointAndResumeWithNoClaim}} fails
> on AZP with
> {code}
> 2021-12-23T03:10:26.4240179Z Dec 23 03:10:26 [ERROR]
> org.apache.flink.test.checkpointing.SavepointITCase.testTriggerSavepointAndResumeWithNoClaim
> Time elapsed: 62.289 s <<< ERROR!
> 2021-12-23T03:10:26.4240998Z Dec 23 03:10:26
> java.util.concurrent.TimeoutException: Condition was not met in given timeout.
> 2021-12-23T03:10:26.4241716Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:166)
> 2021-12-23T03:10:26.4242643Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:144)
> 2021-12-23T03:10:26.4243295Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:136)
> 2021-12-23T03:10:26.4244433Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:210)
> 2021-12-23T03:10:26.4245166Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:184)
> 2021-12-23T03:10:26.4245830Z Dec 23 03:10:26 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:172)
> 2021-12-23T03:10:26.4246870Z Dec 23 03:10:26 at
> org.apache.flink.test.checkpointing.SavepointITCase.testTriggerSavepointAndResumeWithNoClaim(SavepointITCase.java:446)
> 2021-12-23T03:10:26.4247813Z Dec 23 03:10:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-12-23T03:10:26.4248808Z Dec 23 03:10:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-12-23T03:10:26.4249426Z Dec 23 03:10:26 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-23T03:10:26.4250192Z Dec 23 03:10:26 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-12-23T03:10:26.4251196Z Dec 23 03:10:26 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-23T03:10:26.4252160Z Dec 23 03:10:26 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-23T03:10:26.4252888Z Dec 23 03:10:26 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-23T03:10:26.4253547Z Dec 23 03:10:26 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-23T03:10:26.4254142Z Dec 23 03:10:26 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-12-23T03:10:26.4254932Z Dec 23 03:10:26 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-12-23T03:10:26.4255513Z Dec 23 03:10:26 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-12-23T03:10:26.4256091Z Dec 23 03:10:26 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-23T03:10:26.4256636Z Dec 23 03:10:26 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-23T03:10:26.4257165Z Dec 23 03:10:26 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * b0da7ee5efc7652b302161d8eab7989a29e5160c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * b0da7ee5efc7652b302161d8eab7989a29e5160c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
MartijnVisser commented on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1003414058 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25499) Column 'window_start' is ambiguous
[
https://issues.apache.org/jira/browse/FLINK-25499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shandy updated FLINK-25499:
---
Description:
*For docs: [Window
Aggregation|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/queries/window-agg/#cascading-window-aggregation]*
*use sql-client create view such as:*
=
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT a.window_start,a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId) as
tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
group by b.window_start, b.window_end,b.window_time;
=
*above select occurs error message:*
{color:#ff}*[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Column 'window_start' is
ambiguous
*{color}
*if modify create sql like this :*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}-a.windw_start,-{color}a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*or*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}cast(a.window_start as timestamp) as
windowStart,cast(a.window_end as timestamp) as windowEnd,{color}a.window_time
as rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*then, above select-sql can be executed ok!*
was:
*use sql-client create view such as:*
=
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT a.window_start,a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId) as
tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
group by b.window_start, b.window_end,b.window_time;
=
*above select occurs error message:*
{color:#ff}*[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Column 'window_start' is
ambiguous
*{color}
*if modify create sql like this :*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}-a.windw_start,-{color}a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*or*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}cast(a.window_start as timestamp) as
windowStart,cast(a.window_end as timestamp) as windowEnd,{color}a.window_time
as rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*then, above select-sql can be executed ok!*
> Column 'window_start' is ambiguous
> --
>
> Key: FLINK-25499
> URL: https://issues.apache.org/jira/browse/FLINK-25499
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.2
> Environment: Flink 1.14.0
>Reporter: Shandy
>Priority: Major
> Labels: ambiguous, window_start
>
> *For docs: [Window
> Aggregation|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/queries/window-agg/#cascading-window-aggregation]*
> *use sql-client create view such as:*
> =
> CREATE
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * e69e34284d814113503c99bfebac6a01ab5c360f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28814) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25499) Column 'window_start' is ambiguous
[
https://issues.apache.org/jira/browse/FLINK-25499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shandy updated FLINK-25499:
---
Description:
*use sql-client create view such as:*
=
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT a.window_start,a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId) as
tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
group by b.window_start, b.window_end,b.window_time;
=
*above select occurs error message:*
{color:#ff}*[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Column 'window_start' is
ambiguous
*{color}
*if modify create sql like this :*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}-a.windw_start,-{color}a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*or*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}cast(a.window_start as timestamp) as
windowStart,cast(a.window_end as timestamp) as windowEnd,{color}a.window_time
as rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*then, above select-sql can be executed ok!*
was:
*use sql-client create view such as:*
=
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT a.window_start,a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId) as
tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
group by b.window_start, b.window_end,b.window_time;
=
*above select occurs error message:*
{color:#FF}*[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Column 'window_start' is
ambiguous
*{color}
*if modify create sql to({color:#de350b}remove a.window_start{color}) :*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}-a.windw_start,-{color}a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*above select execute success!*
> Column 'window_start' is ambiguous
> --
>
> Key: FLINK-25499
> URL: https://issues.apache.org/jira/browse/FLINK-25499
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.2
> Environment: Flink 1.14.0
>Reporter: Shandy
>Priority: Major
> Labels: ambiguous, window_start
>
> *use sql-client create view such as:*
> =
> CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
> AS
> (
> SELECT a.window_start,a.window_end,a.window_time as
> rowTime,last_value(a.tenantId) as tenantId
> FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
> DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
> group by a.window_start, a.window_end,a.window_time
> );
> SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId)
> as tenantId
> FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
> DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
> group by b.window_start, b.window_end,b.window_time;
> =
> *above select occurs error message:*
> {color:#ff}*[ERROR]
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * acacbdb9f3d5b8eda3d23c9f4339064d8fd43c67 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28813) * 29da74c1ab8840f9a749b76f12b2aa8017d228cd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28815) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * 415cd6722e77556268ff1724ecf60fbb7986b67f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28760) * acacbdb9f3d5b8eda3d23c9f4339064d8fd43c67 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28813) * 29da74c1ab8840f9a749b76f12b2aa8017d228cd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28815) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * 415cd6722e77556268ff1724ecf60fbb7986b67f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28760) * acacbdb9f3d5b8eda3d23c9f4339064d8fd43c67 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28813) * 29da74c1ab8840f9a749b76f12b2aa8017d228cd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
flinkbot edited a comment on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003334073 ## CI report: * 7c99259966c15aaecb655d8c4a00f2217f52dbe6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28810) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18248: [FLINK-25477][docs] Make the directory structure of the State Backend…
flinkbot edited a comment on pull request #18248: URL: https://github.com/apache/flink/pull/18248#issuecomment-1003300867 ## CI report: * a688ff0280c11e838ae6d75fd0c152d084bf72bc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28809) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18169: add shutdown hook
flinkbot edited a comment on pull request #18169: URL: https://github.com/apache/flink/pull/18169#issuecomment-998925832 ## CI report: * 473681213b08ac2dc58ad13f8490542d777f3915 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28811) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25296) [state.checkpoints.num-retained ]The default value does not take effect
[
https://issues.apache.org/jira/browse/FLINK-25296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467258#comment-17467258
]
Piotr Nowojski edited comment on FLINK-25296 at 12/31/21, 2:15 PM:
---
Hi [~libra_816]. What do you mean by {{-Dstate.cache.num-retained}}? I couldn't
find such property/option defined/documented anywhere. Am I missing something?
Did you mean adding {{-Dstate.checkpoints.num-retained}}? If so, I'm also not
sure where are you passing such property, because as far as I know Flink only
supports passing configuration options via {{flink-conf.yaml}} file (please
check [the
docs|https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/]),
or via {{FLINK_PROPERTIES}} on docker-based deployments. Or maybe I am missing
something?
Also for the future, I would suggest reaching out via user mailing list with
questions like that first:
https://flink.apache.org/community.html#mailing-lists
It usually offers faster response times.
was (Author: pnowojski):
Hi [~libra_816]. What do you mean by {{-Dstate.cache.num-retained}}? I couldn't
find such property/option defined/documented anywhere. Am I missing something?
Did you mean adding {{-Dstate.checkpoints.num-retained}}? If so, I'm also not
sure where are you passing such property, because as far as I know Flink only
supports passing configuration options via {{flink-conf.yaml}} file (please
check [the
docs|https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/]),
or via {{FLINK_PROPERTIES}} on docker-based deployments.
Also for the future, I would suggest reaching out via user mailing list with
questions like that first:
https://flink.apache.org/community.html#mailing-lists
It usually offers faster response times.
> [state.checkpoints.num-retained ]The default value does not take effect
> ---
>
> Key: FLINK-25296
> URL: https://issues.apache.org/jira/browse/FLINK-25296
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.3
>Reporter: chenqizhu
>Priority: Major
>
> The configration is valid only when I add -Dstate.cache. num-retained=n.
> If I do not specify this configuration, the default value does not take
> effect , is it a bug ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25296) [state.checkpoints.num-retained ]The default value does not take effect
[
https://issues.apache.org/jira/browse/FLINK-25296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467258#comment-17467258
]
Piotr Nowojski commented on FLINK-25296:
Hi [~libra_816]. What do you mean by {{-Dstate.cache.num-retained}}? I couldn't
find such property/option defined/documented anywhere. Am I missing something?
Did you mean adding {{-Dstate.checkpoints.num-retained}}? If so, I'm also not
sure where are you passing such property, because as far as I know Flink only
supports passing configuration options via {{flink-conf.yaml}} file (please
check [the
docs|https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/]),
or via {{FLINK_PROPERTIES}} on docker-based deployments.
Also for the future, I would suggest reaching out via user mailing list with
questions like that first:
https://flink.apache.org/community.html#mailing-lists
It usually offers faster response times.
> [state.checkpoints.num-retained ]The default value does not take effect
> ---
>
> Key: FLINK-25296
> URL: https://issues.apache.org/jira/browse/FLINK-25296
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.3
>Reporter: chenqizhu
>Priority: Major
>
> The configration is valid only when I add -Dstate.cache. num-retained=n.
> If I do not specify this configuration, the default value does not take
> effect , is it a bug ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * b0da7ee5efc7652b302161d8eab7989a29e5160c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18234: [hotfix][docs] Fixing multiple 404 links in docs
flinkbot edited a comment on pull request #18234: URL: https://github.com/apache/flink/pull/18234#issuecomment-1002540809 ## CI report: * ac76fbad4c7ea76fbacec96d1921ae50d53aae8b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28805) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18250: [hotfix][docs] Remove duplicate dot in generating_watermarks.md
flinkbot edited a comment on pull request #18250: URL: https://github.com/apache/flink/pull/18250#issuecomment-1003346407 ## CI report: * a92d35de7823a738ece605bbd49df88bceb48c5c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28804) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17847: [FLINK-6573][Connectors/Common ] Flink MongoDB Connector
flinkbot edited a comment on pull request #17847: URL: https://github.com/apache/flink/pull/17847#issuecomment-974817636 ## CI report: * bfaf00877a5a850d3e85eee6bb22c59990f351da Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28812) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * b86b30a505124aa2404eb1e0d6178eaf600d6368 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28796) * e69e34284d814113503c99bfebac6a01ab5c360f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28814) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * b86b30a505124aa2404eb1e0d6178eaf600d6368 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28796) * e69e34284d814113503c99bfebac6a01ab5c360f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * 415cd6722e77556268ff1724ecf60fbb7986b67f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28760) * acacbdb9f3d5b8eda3d23c9f4339064d8fd43c67 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28813) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #18014: [FLINK-24857][test][Kafka] Upgrade SourceReaderTestBase t…
imaffe commented on pull request #18014: URL: https://github.com/apache/flink/pull/18014#issuecomment-1003372339 Sure, have a great vacation and happy new year everyone ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18181: [FLINK-25414][metrics] Provide metrics to measure how long task has been blocked
flinkbot edited a comment on pull request #18181: URL: https://github.com/apache/flink/pull/18181#issuecomment-999589676 ## CI report: * 415cd6722e77556268ff1724ecf60fbb7986b67f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28760) * acacbdb9f3d5b8eda3d23c9f4339064d8fd43c67 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25499) Column 'window_start' is ambiguous
Shandy created FLINK-25499:
--
Summary: Column 'window_start' is ambiguous
Key: FLINK-25499
URL: https://issues.apache.org/jira/browse/FLINK-25499
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.14.2
Environment: Flink 1.14.0
Reporter: Shandy
*use sql-client create view such as:*
=
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT a.window_start,a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
SELECT b.window_start, b.window_end,b.window_time as rowTime,sum(b.tenantId) as
tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_standard_t,
DESCRIPTOR(rowTime), INTERVAL '60' MINUTES)) as b
group by b.window_start, b.window_end,b.window_time;
=
*above select occurs error message:*
{color:#FF}*[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Column 'window_start' is
ambiguous
*{color}
*if modify create sql to({color:#de350b}remove a.window_start{color}) :*
CREATE TEMPORARY VIEW IF NOT EXISTS telemetry_r_yangchen_standard_t
AS
(
SELECT {color:#de350b}-a.windw_start,-{color}a.window_end,a.window_time as
rowTime,last_value(a.tenantId) as tenantId
FROM TABLE(TUMBLE(TABLE telemetry_r_yangchen_normal,
DESCRIPTOR(receiveTimeTS), INTERVAL '10' MINUTES)) as a
group by a.window_start, a.window_end,a.window_time
);
*above select execute success!*
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-22643) Too many TCP connections among TaskManagers for large scale jobs
[
https://issues.apache.org/jira/browse/FLINK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467241#comment-17467241
]
fanrui commented on FLINK-22643:
[~pnowojski] Thanks for your suggestion.
I will ensure that each task manager having only a single task slot. If there
are enough resources, I will test the job with higher parallelism.
> Too many TCP connections among TaskManagers for large scale jobs
>
>
> Key: FLINK-22643
> URL: https://issues.apache.org/jira/browse/FLINK-22643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Zhilong Hong
>Assignee: fanrui
>Priority: Minor
> Labels: auto-deprioritized-major
>
> For the large scale jobs, there will be too many TCP connections among
> TaskManagers. Let's take an example.
> For a streaming job with 20 JobVertices, each JobVertex has 500 parallelism.
> We divide the vertices into 5 slot sharing groups. Each TaskManager has 5
> slots. Thus there will be 400 taskmanagers in this job. Let's assume that job
> runs on a cluster with 20 machines.
> If all the job edges are all-to-all edges, there will be 19 * 20 * 399 * 2 =
> 303,240 TCP connections for each machine. If we run several jobs on this
> cluster, the TCP connections may exceed the maximum limit of linux, which is
> 1,048,576. This will stop the TaskManagers from creating new TCP connections
> and cause task failovers.
> As we run our production jobs on a K8S cluster, the job always failover due
> to exceptions related to network, such as {{Sending the partition request to
> 'null' failed}}, and etc.
> We think that we can decrease the number of connections by letting tasks
> reuse the same connection. We implemented a POC that makes all tasks on the
> same TaskManager reuse one TCP connection. For the example job we mentioned
> above, the number of connections will decrease from 303,240 to 15960. With
> the POC, the frequency of meeting exceptions related to network in our
> production jobs drops significantly.
> The POC is illustrated in:
> https://github.com/wsry/flink/commit/bf1c09e80450f40d018a1d1d4fe3dfd2de777fdc
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #14858: [FLINK-16154] Translate "Operator/Join" into Chinese
flinkbot edited a comment on pull request #14858: URL: https://github.com/apache/flink/pull/14858#issuecomment-773005730 ## CI report: * 378e94317d27cba26ef3021c22f5f0c81e93fec8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28802) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17847: [FLINK-6573][Connectors/Common ] Flink MongoDB Connector
flinkbot edited a comment on pull request #17847: URL: https://github.com/apache/flink/pull/17847#issuecomment-974817636 ## CI report: * e8d0655fe2102a599057673f317cf214a2e41447 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28670) * bfaf00877a5a850d3e85eee6bb22c59990f351da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28812) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18169: add shutdown hook
flinkbot edited a comment on pull request #18169: URL: https://github.com/apache/flink/pull/18169#issuecomment-998925832 ## CI report: * 42808a2af4ff41522881e8ca9c02262b995ae87b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28795) * 473681213b08ac2dc58ad13f8490542d777f3915 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28811) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17847: [FLINK-6573][Connectors/Common ] Flink MongoDB Connector
flinkbot edited a comment on pull request #17847: URL: https://github.com/apache/flink/pull/17847#issuecomment-974817636 ## CI report: * e8d0655fe2102a599057673f317cf214a2e41447 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28670) * bfaf00877a5a850d3e85eee6bb22c59990f351da UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18169: add shutdown hook
flinkbot edited a comment on pull request #18169: URL: https://github.com/apache/flink/pull/18169#issuecomment-998925832 ## CI report: * 42808a2af4ff41522881e8ca9c02262b995ae87b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28795) * 473681213b08ac2dc58ad13f8490542d777f3915 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
flinkbot edited a comment on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003334073 ## CI report: * Unknown: [CANCELED](TBD) * 7c99259966c15aaecb655d8c4a00f2217f52dbe6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28810) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuzifu666 commented on pull request #18243: [MINOR] remove unused LOG reference
xuzifu666 commented on pull request #18243: URL: https://github.com/apache/flink/pull/18243#issuecomment-1003364436 @gaoyunhaii hi! have a review please, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
flinkbot edited a comment on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003334073 ## CI report: * 9a881ffe7c03650e16e2c450ed51888f3c94b560 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28801) * Unknown: [CANCELED](TBD) * 7c99259966c15aaecb655d8c4a00f2217f52dbe6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28810) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18033: [FLINK-25141][connector/elasticsearch][docs] Add sink parallelism option
flinkbot edited a comment on pull request #18033: URL: https://github.com/apache/flink/pull/18033#issuecomment-987520681 ## CI report: * 4b12ff1669fe0c86bf3f29939520aa3efebcc555 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28799) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr closed pull request #18162: [FLINK-25187][table-planner] Apply padding when CASTing to BINARY()
twalthr closed pull request #18162: URL: https://github.com/apache/flink/pull/18162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
flinkbot edited a comment on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003334073 ## CI report: * 9a881ffe7c03650e16e2c450ed51888f3c94b560 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28801) * Unknown: [CANCELED](TBD) * 7c99259966c15aaecb655d8c4a00f2217f52dbe6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18248: [FLINK-25477][docs] Make the directory structure of the State Backend…
flinkbot edited a comment on pull request #18248: URL: https://github.com/apache/flink/pull/18248#issuecomment-1003300867 ## CI report: * 5a11d1fbfe8fcc416bd6f827d49b9576eba76c76 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28792) * a688ff0280c11e838ae6d75fd0c152d084bf72bc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28809) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14773: [FLINK-16153] Translate "Operator/windows" into Chinese
flinkbot edited a comment on pull request #14773: URL: https://github.com/apache/flink/pull/14773#issuecomment-768234436 ## CI report: * aa11c3a56df1220308dc2d6fb6ce9371eb29d684 UNKNOWN * 7dca580601088ab159a5a4b0df01511844f383d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28798) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zuston commented on pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
zuston commented on pull request #18249: URL: https://github.com/apache/flink/pull/18249#issuecomment-1003359381 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18248: [FLINK-25477][docs] Make the directory structure of the State Backend…
flinkbot edited a comment on pull request #18248: URL: https://github.com/apache/flink/pull/18248#issuecomment-1003300867 ## CI report: * 5a11d1fbfe8fcc416bd6f827d49b9576eba76c76 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28792) * a688ff0280c11e838ae6d75fd0c152d084bf72bc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16609: [FLINK-23324][connector/jdbc] Make Postgres case-sensitive.
flinkbot edited a comment on pull request #16609: URL: https://github.com/apache/flink/pull/16609#issuecomment-887558020 ## CI report: * 137482a836aa96da49f690b27275f10034b41681 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28797) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25187) Apply padding for BINARY()
[
https://issues.apache.org/jira/browse/FLINK-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-25187.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master: 49acb2723eda8ebd3fc59af19d4bc0abb9f1a318
> Apply padding for BINARY()
> -
>
> Key: FLINK-25187
> URL: https://issues.apache.org/jira/browse/FLINK-25187
> Project: Flink
> Issue Type: Sub-task
>Reporter: Marios Trivyzas
>Assignee: Marios Trivyzas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> When the resulting byte array that is generated for a *CAST(XXX AS
> BINARY()* has *length* < {*}precision{*}, then it should be padded
> with *0* to the right, to end up with a byte array of *precision* length,
> similarly to padding with spaces for {*}CHAR(){*}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zuston commented on a change in pull request #18249: [FLINK-25496] Fix the compatibility issue of jdk8 and 11 in ThreadDum…
zuston commented on a change in pull request #18249:
URL: https://github.com/apache/flink/pull/18249#discussion_r776990985
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/rest/messages/ThreadDumpInfoTest.java
##
@@ -86,4 +94,12 @@ public void testStacktraceDepthLimitation() {
private long getOutputDepth(String stringifiedInfo) {
return Arrays.stream(stringifiedInfo.split("\n")).filter(x ->
x.contains("\tat ")).count();
}
+
+private String[] removeFirstIndex(String[] lines) {
+String[] remained = new String[lines.length - 1];
Review comment:
Of course! I forgot this method. Thanks for pointing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-21186) RecordWriterOutput swallows interrupt state when interrupted.
[ https://issues.apache.org/jira/browse/FLINK-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-21186. -- Fix Version/s: 1.15.0 Resolution: Fixed Merged a small hotfix change as commit ef839ff into apache:master > RecordWriterOutput swallows interrupt state when interrupted. > - > > Key: FLINK-21186 > URL: https://issues.apache.org/jira/browse/FLINK-21186 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.13.0 >Reporter: Arvid Heise >Assignee: Piotr Nowojski >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] pnowojski merged pull request #18159: [FLINK-21186][network] Wrap IOException in UncheckedIOException in RecordWriterOutput
pnowojski merged pull request #18159: URL: https://github.com/apache/flink/pull/18159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * Unknown: [CANCELED](TBD) * b0da7ee5efc7652b302161d8eab7989a29e5160c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18209: [FLINK-25461][python] Update net.sf.py4j:py4j dependency to 0.10.9.3
flinkbot edited a comment on pull request #18209: URL: https://github.com/apache/flink/pull/18209#issuecomment-1001578474 ## CI report: * 611af0ce8e2999f7896a259edab03b8d1103eb1c UNKNOWN * Unknown: [CANCELED](TBD) * b0da7ee5efc7652b302161d8eab7989a29e5160c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
pnowojski commented on a change in pull request #18068: URL: https://github.com/apache/flink/pull/18068#discussion_r776988263 ## File path: docs/content/docs/dev/datastream/fault-tolerance/checkpointing.md ## @@ -271,4 +271,8 @@ Any operator that is prepared to be rescaled should work well with tasks that pa Restoring from a checkpoint where only a subset of tasks finished is equivalent to restoring such a task with the number of new subtasks equal to the number of finished tasks. +### Waiting for the final checkpoint before task exit + +To ensure all the records could be committed for operators using the two-phase commit, the tasks would wait for the final checkpoint completed successfully after all the operators finished. It needs to be noted that this behavior would prolong the execution time of tasks. If the checkpoint interval is long, the execution time would also be prolonged largely. For the worst case, if the checkpoint interval is set to `Long.MAX_VALUE`, the tasks would in fact be blocked forever since the final checkpoint would never happen. Review comment: ``` To ensure all the records will be committed for operators using the two-phase commit, the subtasks have to wait for the final checkpoint to complete successfully after subtask finished processing new records. It is worth noting that this behavior prolongs the execution time of subtasks. If either the checkpoint interval is long or the end to end checkpoint duration is long, the execution time will be prolonged significantly. In the worst case, if the checkpoint interval is set to `Long.MAX_VALUE`, the tasks would in fact be blocked forever since the final checkpoint will never happen. ``` + can you split it into multiple lines? Docs that have paragraphs in a single line are causing quite a huge mess when reviewing the diffs or resolving merge conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
