[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats
[ https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512400#comment-15512400 ] ASF GitHub Bot commented on FLINK-3719: --- Github user mushketyk commented on the issue: https://github.com/apache/flink/pull/2467 Hi @StephanEwen All images are coming from this repo: https://github.com/nathancahill/Split.js It has a very permissive license: https://github.com/nathancahill/Split.js/blob/master/LICENSE.txt > WebInterface: Moving the barrier between graph and stats > > > Key: FLINK-3719 > URL: https://issues.apache.org/jira/browse/FLINK-3719 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: Niels Basjes >Assignee: Ivan Mushketyk > > It would be really useful if the separator between the graphical view of a > job topology at the top and the textual overview of the counters at the > bottom can be moved up/down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2533 [FLINK-4603] Fixes: KeyedStateBackend cannot restore user code classes This PR fixes [FLINK-4603] and introduces a test to protect better against future regression. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StefanRRichter/flink backend-classloader-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2533.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2533 commit d6b8b0112c6a4cf3f2cbf5eb758599e15d796aab Author: Stefan Richter Date: 2016-09-21T12:55:58Z [FLINK-4603] KeyedStateBackend can restore user code classes commit 78b2a4f048bd62e55471a384169304ca46bbbf60 Author: Stefan Richter Date: 2016-09-21T15:56:08Z [FLINK-4603] Test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2533 Please review @tillrohrmann or @aljoscha --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512540#comment-15512540
]
ASF GitHub Bot commented on FLINK-4603:
---
GitHub user StefanRRichter opened a pull request:
https://github.com/apache/flink/pull/2533
[FLINK-4603] Fixes: KeyedStateBackend cannot restore user code classes
This PR fixes [FLINK-4603] and introduces a test to protect better against
future regression.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StefanRRichter/flink backend-classloader-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2533.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2533
commit d6b8b0112c6a4cf3f2cbf5eb758599e15d796aab
Author: Stefan Richter
Date: 2016-09-21T12:55:58Z
[FLINK-4603] KeyedStateBackend can restore user code classes
commit 78b2a4f048bd62e55471a384169304ca46bbbf60
Author: Stefan Richter
Date: 2016-09-21T15:56:08Z
[FLINK-4603] Test case
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512541#comment-15512541
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/2533
Please review @tillrohrmann or @aljoscha
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512553#comment-15512553
]
Stefan Richter commented on FLINK-4603:
---
Currently, we should still keep the UserCodeClassLoader around in the RocksDB
backend because we still need to serialize the StateDescriptor, which contains
the TypeSerializer, so that users can not accidentally create StateDescriptors
with a wrong TypeSerializer. However, we should consider that TypeSerializer
can be exchanged (ensuring their compatibility), e.g. to allow different
serialization versions.
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4535) ResourceManager registration with TaskExecutor
[ https://issues.apache.org/jira/browse/FLINK-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512579#comment-15512579 ] ASF GitHub Bot commented on FLINK-4535: --- Github user mxm commented on the issue: https://github.com/apache/flink/pull/2451 This has been merged to `flip-6`. Could you please close the PR? > ResourceManager registration with TaskExecutor > -- > > Key: FLINK-4535 > URL: https://issues.apache.org/jira/browse/FLINK-4535 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing >Assignee: zhangjing > > When TaskExecutor register at ResourceManager, it takes the following 3 input > parameters: > 1. resourceManagerLeaderId: the fencing token for the ResourceManager leader > which is kept by taskExecutor who send the registration > 2. taskExecutorAddress: the address of taskExecutor > 3. resourceID: The resource ID of the TaskExecutor that registers > ResourceManager need to process the registration event based on the following > steps: > 1. Check whether input resourceManagerLeaderId is as same as the current > leadershipSessionId of resourceManager. If not, it means that maybe two or > more resourceManager exists at the same time, and current resourceManager is > not the proper rm. so it rejects or ignores the registration. > 2. Check whether exists a valid taskExecutor at the giving address by > connecting to the address. Reject the registration from invalid address. > 3. Check whether it is a duplicate registration by input resourceId, reject > the registration > 4. Keep resourceID and taskExecutorGateway mapping relationships, And > optionally keep resourceID and container mapping relationships in yarn mode. > 5. Create the connection between resourceManager and taskExecutor, and ensure > its healthy based on heartbeat rpc calls between rm and tm ? > 6. Send registration successful ack to the taskExecutor. > Discussion: > Maybe we need import errorCode or several registration decline subclass to > distinguish the different causes of decline registration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2479: [FLINK-4537] [cluster management] ResourceManager registr...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2479 This has been merged to `flip-6`. Could you please close the PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2451: [FLINK-4535] [cluster management] resourceManager process...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2451 This has been merged to `flip-6`. Could you please close the PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4537) ResourceManager registration with JobManager
[
https://issues.apache.org/jira/browse/FLINK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512578#comment-15512578
]
ASF GitHub Bot commented on FLINK-4537:
---
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2479
This has been merged to `flip-6`. Could you please close the PR?
> ResourceManager registration with JobManager
>
>
> Key: FLINK-4537
> URL: https://issues.apache.org/jira/browse/FLINK-4537
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Maximilian Michels
>Assignee: zhangjing
>
> The ResourceManager keeps tracks of all JobManager's which execute Jobs. When
> a new JobManager registered, its leadership status is checked through the
> HighAvailabilityServices. It will then be registered at the ResourceManager
> using the {{JobID}} provided with the initial registration message.
> ResourceManager should use JobID and LeaderSessionID(notified by
> HighAvailabilityServices) to identify a a session to JobMaster.
> When JobManager's register at ResourceManager, it takes the following 2 input
> parameters :
> 1. resourceManagerLeaderId: the fencing token for the ResourceManager leader
> which is kept by JobMaster who send the registration
> 2. JobMasterRegistration: contain address, JobID
> ResourceManager need to process the registration event based on the following
> steps:
> 1. Check whether input resourceManagerLeaderId is as same as the current
> leadershipSessionId of resourceManager. If not, it means that maybe two or
> more resourceManager exists at the same time, and current resourceManager is
> not the proper rm. so it rejects or ignores the registration.
> 2. Check whether exists a valid JobMaster at the giving address by connecting
> to the address. Reject the registration from invalid address.(Hidden in the
> connect logic)
> 3. Keep JobID and JobMasterGateway mapping relationships.
> 4. Start a JobMasterLeaderListener at the given JobID to listen to the
> leadership of the specified JobMaster.
> 5. Send registration successful ack to the jobMaster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4311) TableInputFormat fails when reused on next split
[
https://issues.apache.org/jira/browse/FLINK-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512604#comment-15512604
]
ASF GitHub Bot commented on FLINK-4311:
---
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2330
Hi @nielsbasjes, thanks for fixing and cleaning up the `TableInputFormat`.
This PR is good to merge.
> TableInputFormat fails when reused on next split
>
>
> Key: FLINK-4311
> URL: https://issues.apache.org/jira/browse/FLINK-4311
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.3
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Critical
>
> We have written a batch job that uses data from HBase by means of using the
> TableInputFormat.
> We have found that this class sometimes fails with this exception:
> {quote}
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
> Task
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
> rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:152)
> at
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:47)
> at
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:147)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.RejectedExecutionException: Task
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4f4efe4b
> rejected from java.util.concurrent.ThreadPoolExecutor@7872d5c1[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1165]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:142)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:269)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:165)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> ... 10 more
> {quote}
> As you can see the ThreadPoolExecutor was terminated at this point.
> We tracked it down to the fact that
> # the configure method opens the table
> # the open method obtains the result scanner
> # the closes method closes the table.
> If a second split arrives on the same instance then the open method will fail
> because the table has already been closed.
> We also found that this error varies with the versions of HBase that are
> used. I have also seen this exception:
> {quote}
> Caused by: java.io.IOException: hconnection-0x19d37183 closed
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> ... 37 more
> {quote}
> I found that in the [documentation of the InputFormat
> interface|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/io/InputFormat.html]
> is clearly states
> {quote}IMPORTANT NOTE: Input formats must be written such that an instance
> can be opened again after it was closed. That is due to the fact that the
> input format is used for potentially multiple splits. After a split is done,
> the format's close function is invoked and, if another split is available,
> the open function is invoked afterwa
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 Hi @nielsbasjes, thanks for fixing and cleaning up the `TableInputFormat`. This PR is good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter
[
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512621#comment-15512621
]
ASF GitHub Bot commented on FLINK-4564:
---
Github user ex00 commented on the issue:
https://github.com/apache/flink/pull/2517
Could you explain me how must look API for getting metric identifier?
Now I think about this:
```java
@Test
public void testConfigurableDelimiterForReporter() {
Configuration config = new Configuration();
config.setString(ConfigConstants.METRICS_REPORTERS_LIST,
"test1");
config.setString(ConfigConstants.METRICS_SCOPE_DELIMITER, ".");
config.setString(ConfigConstants.METRICS_REPORTER_PREFIX +
"test1." + ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER, "_");
config.setString(ConfigConstants.METRICS_REPORTER_PREFIX +
"test1." + ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
TestReporter.class.getName());
config.setString(ConfigConstants.METRICS_SCOPE_NAMING_TM,
"A.B.C.D.E");
MetricRegistry registry = new MetricRegistry(config);
TaskManagerMetricGroup tmGroup = new
TaskManagerMetricGroup(registry, "host", "id");
assertEquals("A.B.C.D.E.name",
tmGroup.getMetricIdentifier("name")); //get default delimiter for all
assertEquals("A_B_C_D_E_name",
tmGroup.getMetricIdentifier("name",null,1)); //get delimiter for first reporter
in list
registry.shutdown();
}
```
> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
> Issue Type: Bug
> Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration
> setting shared by all reporters. However, different reporters may have
> different requirements in regards to the delimiter, as such we should allow
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no
> specific setting was set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2517: [FLINK-4564] [metrics] Delimiter should be configured per...
Github user ex00 commented on the issue:
https://github.com/apache/flink/pull/2517
Could you explain me how must look API for getting metric identifier?
Now I think about this:
```java
@Test
public void testConfigurableDelimiterForReporter() {
Configuration config = new Configuration();
config.setString(ConfigConstants.METRICS_REPORTERS_LIST,
"test1");
config.setString(ConfigConstants.METRICS_SCOPE_DELIMITER, ".");
config.setString(ConfigConstants.METRICS_REPORTER_PREFIX +
"test1." + ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER, "_");
config.setString(ConfigConstants.METRICS_REPORTER_PREFIX +
"test1." + ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
TestReporter.class.getName());
config.setString(ConfigConstants.METRICS_SCOPE_NAMING_TM,
"A.B.C.D.E");
MetricRegistry registry = new MetricRegistry(config);
TaskManagerMetricGroup tmGroup = new
TaskManagerMetricGroup(registry, "host", "id");
assertEquals("A.B.C.D.E.name",
tmGroup.getMetricIdentifier("name")); //get default delimiter for all
assertEquals("A_B_C_D_E_name",
tmGroup.getMetricIdentifier("name",null,1)); //get delimiter for first reporter
in list
registry.shutdown();
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2459: [FLINK-4561] replace all the scala version as a `scala.bi...
Github user shijinkui commented on the issue: https://github.com/apache/flink/pull/2459 hi, @StephanEwen I have an idea in mind, If there are to much tricky bugs, and follow "don't change it unless it is broken", after two years can you image the code like? Full of tricky bugs fix. I think we should make the code robust, reasonable and readable. Can we make a long plan with many sub-tasks to reach the above purpose? Such as reduce the unnecessary module, pom more simple and easy maintain the pom.xml Many people consider the Flink code confused and then give up Flink. I think Flink's features in whole are better, the detail of project need enhance. How do you think about that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4561) replace all the scala version as a `scala.binary.version` property
[ https://issues.apache.org/jira/browse/FLINK-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512657#comment-15512657 ] ASF GitHub Bot commented on FLINK-4561: --- Github user shijinkui commented on the issue: https://github.com/apache/flink/pull/2459 hi, @StephanEwen I have an idea in mind, If there are to much tricky bugs, and follow "don't change it unless it is broken", after two years can you image the code like? Full of tricky bugs fix. I think we should make the code robust, reasonable and readable. Can we make a long plan with many sub-tasks to reach the above purpose? Such as reduce the unnecessary module, pom more simple and easy maintain the pom.xml Many people consider the Flink code confused and then give up Flink. I think Flink's features in whole are better, the detail of project need enhance. How do you think about that? > replace all the scala version as a `scala.binary.version` property > -- > > Key: FLINK-4561 > URL: https://issues.apache.org/jira/browse/FLINK-4561 > Project: Flink > Issue Type: Improvement >Reporter: shijinkui > > replace all the scala version(2.10) as a property `scala.binary.version` > defined in root pom properties. default scala version property is 2.10. > modify: > 1. dependency include scala version > 2. module defining include scala version > 3. scala version upgrade to 2.11.8 from 2.11.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2533 Fix looks good. Better would probably be to not even have the user code in the checkpoint at all. Can we do that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512702#comment-15512702
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2533
Fix looks good.
Better would probably be to not even have the user code in the checkpoint
at all.
Can we do that?
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2303: [FLINK-4248] [core] [table] CsvTableSource does no...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512709#comment-15512709
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2303
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther resolved FLINK-4248.
-
Resolution: Fixed
Fix Version/s: 1.2.0
Fixed in 3507d59f969485dd735487e6bf3eb893b2e3d8ed.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
> Fix For: 1.2.0
>
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
shijinkui created FLINK-4661: Summary: Failure to find org.apache.flink:flink-runtime_2.10:jar:tests Key: FLINK-4661 URL: https://issues.apache.org/jira/browse/FLINK-4661 Project: Flink Issue Type: Bug Reporter: shijinkui [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could not resolve dependencies for project org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in http://localhost:/repository/maven-public/ was cached in the local repository, resolution will not be reattempted until the update interval of nexus-releases has elapsed or updates are forced -> [Help 1] Failure to find org.apache.flink:flink-runtime_2.10:jar:tests I can't find where this tests jar is generated. By the way, recently half month, I start to use flink. There is zero time I can compile the Flink project with default setting.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2533 @StephanEwen we can do that but then we won't have any sanity checks for the `TypeSerializer` any more. Right now, even the RocksDB backed will serializer the `TypeSerializer`/`StateDescriptor` with the checkpoint to verify that the user only accesses it with the correct `TypeSerializer`/`StateDescriptor`. I would be in favor of completely getting rid of user code there, even if it means losing those checks. Also, for this to work with the Heap backend we need to either always keep state on the heap in serialized form or deserialize lazily from restored serialized values using the `TypeSerializer` that we get from the user when they access state for the first time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512765#comment-15512765
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2533
@StephanEwen we can do that but then we won't have any sanity checks for
the `TypeSerializer` any more. Right now, even the RocksDB backed will
serializer the `TypeSerializer`/`StateDescriptor` with the checkpoint to verify
that the user only accesses it with the correct
`TypeSerializer`/`StateDescriptor`.
I would be in favor of completely getting rid of user code there, even if
it means losing those checks. Also, for this to work with the Heap backend we
need to either always keep state on the heap in serialized form or deserialize
lazily from restored serialized values using the `TypeSerializer` that we get
from the user when they access state for the first time.
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
[ https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shijinkui updated FLINK-4661: - Summary: Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT (was: Failure to find org.apache.flink:flink-runtime_2.10:jar:tests) > Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT > -- > > Key: FLINK-4661 > URL: https://issues.apache.org/jira/browse/FLINK-4661 > Project: Flink > Issue Type: Bug >Reporter: shijinkui > > [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could > not resolve dependencies for project > org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find > org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in > http://localhost:/repository/maven-public/ was cached in the local > repository, resolution will not be reattempted until the update interval of > nexus-releases has elapsed or updates are forced -> [Help 1] > Failure to find org.apache.flink:flink-runtime_2.10:jar:tests > I can't find where this tests jar is generated. > By the way, recently half month, I start to use flink. There is zero time I > can compile the Flink project with default setting.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2533#discussion_r80005604
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
---
@@ -266,18 +265,20 @@ public void
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
- ObjectInputStream ois = new
ObjectInputStream(inView);
+ TypeSerializer namespaceSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+ TypeSerializer stateSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
- TypeSerializer namespaceSerializer =
(TypeSerializer) ois.readObject();
- TypeSerializer stateSerializer =
(TypeSerializer) ois.readObject();
- StateTable stateTable = new
StateTable(stateSerializer,
+ StateTable stateTable = new StateTable(
+ stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
- for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+ for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --
Was this wrong before?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512773#comment-15512773
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2533#discussion_r80005604
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
---
@@ -266,18 +265,20 @@ public void
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
- ObjectInputStream ois = new
ObjectInputStream(inView);
+ TypeSerializer namespaceSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+ TypeSerializer stateSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
- TypeSerializer namespaceSerializer =
(TypeSerializer) ois.readObject();
- TypeSerializer stateSerializer =
(TypeSerializer) ois.readObject();
- StateTable stateTable = new
StateTable(stateSerializer,
+ StateTable stateTable = new StateTable(
+ stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
- for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+ for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --
Was this wrong before?
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4650) Frequent task manager disconnects from JobManager
[ https://issues.apache.org/jira/browse/FLINK-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512786#comment-15512786 ] Stephan Ewen commented on FLINK-4650: - Am I right in assuming that the TaskManager is not marked as lost by the JobManager (disappears in the web UI), but the connection for shuffles breaks. Does it recover properly? Would be good to see if the source node (titus-248496-worker-0-2/100.82.8.187) has anything suspicious in the logs. > Frequent task manager disconnects from JobManager > - > > Key: FLINK-4650 > URL: https://issues.apache.org/jira/browse/FLINK-4650 > Project: Flink > Issue Type: Bug >Reporter: Nagarjun Guraja > > Not sure of the exact reason but we observe more frequent task manager > disconnects while using 1.2 snapshot build as compared to 1.1.2 release build -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2479: [FLINK-4537] [cluster management] ResourceManager ...
Github user beyond1920 closed the pull request at: https://github.com/apache/flink/pull/2479 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4535) ResourceManager registration with TaskExecutor
[ https://issues.apache.org/jira/browse/FLINK-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512796#comment-15512796 ] ASF GitHub Bot commented on FLINK-4535: --- Github user beyond1920 closed the pull request at: https://github.com/apache/flink/pull/2451 > ResourceManager registration with TaskExecutor > -- > > Key: FLINK-4535 > URL: https://issues.apache.org/jira/browse/FLINK-4535 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing >Assignee: zhangjing > > When TaskExecutor register at ResourceManager, it takes the following 3 input > parameters: > 1. resourceManagerLeaderId: the fencing token for the ResourceManager leader > which is kept by taskExecutor who send the registration > 2. taskExecutorAddress: the address of taskExecutor > 3. resourceID: The resource ID of the TaskExecutor that registers > ResourceManager need to process the registration event based on the following > steps: > 1. Check whether input resourceManagerLeaderId is as same as the current > leadershipSessionId of resourceManager. If not, it means that maybe two or > more resourceManager exists at the same time, and current resourceManager is > not the proper rm. so it rejects or ignores the registration. > 2. Check whether exists a valid taskExecutor at the giving address by > connecting to the address. Reject the registration from invalid address. > 3. Check whether it is a duplicate registration by input resourceId, reject > the registration > 4. Keep resourceID and taskExecutorGateway mapping relationships, And > optionally keep resourceID and container mapping relationships in yarn mode. > 5. Create the connection between resourceManager and taskExecutor, and ensure > its healthy based on heartbeat rpc calls between rm and tm ? > 6. Send registration successful ack to the taskExecutor. > Discussion: > Maybe we need import errorCode or several registration decline subclass to > distinguish the different causes of decline registration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2451: [FLINK-4535] [cluster management] resourceManager ...
Github user beyond1920 closed the pull request at: https://github.com/apache/flink/pull/2451 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4537) ResourceManager registration with JobManager
[
https://issues.apache.org/jira/browse/FLINK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512795#comment-15512795
]
ASF GitHub Bot commented on FLINK-4537:
---
Github user beyond1920 closed the pull request at:
https://github.com/apache/flink/pull/2479
> ResourceManager registration with JobManager
>
>
> Key: FLINK-4537
> URL: https://issues.apache.org/jira/browse/FLINK-4537
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Maximilian Michels
>Assignee: zhangjing
>
> The ResourceManager keeps tracks of all JobManager's which execute Jobs. When
> a new JobManager registered, its leadership status is checked through the
> HighAvailabilityServices. It will then be registered at the ResourceManager
> using the {{JobID}} provided with the initial registration message.
> ResourceManager should use JobID and LeaderSessionID(notified by
> HighAvailabilityServices) to identify a a session to JobMaster.
> When JobManager's register at ResourceManager, it takes the following 2 input
> parameters :
> 1. resourceManagerLeaderId: the fencing token for the ResourceManager leader
> which is kept by JobMaster who send the registration
> 2. JobMasterRegistration: contain address, JobID
> ResourceManager need to process the registration event based on the following
> steps:
> 1. Check whether input resourceManagerLeaderId is as same as the current
> leadershipSessionId of resourceManager. If not, it means that maybe two or
> more resourceManager exists at the same time, and current resourceManager is
> not the proper rm. so it rejects or ignores the registration.
> 2. Check whether exists a valid JobMaster at the giving address by connecting
> to the address. Reject the registration from invalid address.(Hidden in the
> connect logic)
> 3. Keep JobID and JobMasterGateway mapping relationships.
> 4. Start a JobMasterLeaderListener at the given JobID to listen to the
> leadership of the specified JobMaster.
> 5. Send registration successful ack to the jobMaster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4646) Add BipartiteGraph class
[ https://issues.apache.org/jira/browse/FLINK-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512828#comment-15512828 ] Stephan Ewen commented on FLINK-4646: - [~greghogan] [~vkalavri] Copying you here. Are these features you are looking to get into Gelly? > Add BipartiteGraph class > > > Key: FLINK-4646 > URL: https://issues.apache.org/jira/browse/FLINK-4646 > Project: Flink > Issue Type: Sub-task > Components: Gelly >Reporter: Ivan Mushketyk > > Implement a class to represent a bipartite graph in Flink Gelly. Design > discussions can be found in the parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/2533#discussion_r80010321
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
---
@@ -266,18 +265,20 @@ public void
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
- ObjectInputStream ois = new
ObjectInputStream(inView);
+ TypeSerializer namespaceSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+ TypeSerializer stateSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
- TypeSerializer namespaceSerializer =
(TypeSerializer) ois.readObject();
- TypeSerializer stateSerializer =
(TypeSerializer) ois.readObject();
- StateTable stateTable = new
StateTable(stateSerializer,
+ StateTable stateTable = new StateTable(
+ stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
- for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+ for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --
I think i wanted to break this because it exceeds the line limit but then
decided against it because IntelliJ messed up the formatting for loops. Nothing
wrong there at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (FLINK-4662) Bump Calcite version up to 1.9
Timo Walther created FLINK-4662: --- Summary: Bump Calcite version up to 1.9 Key: FLINK-4662 URL: https://issues.apache.org/jira/browse/FLINK-4662 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Timo Walther Calcite just released the 1.9 version. We should adopt it also in the Table API especially for FLINK-4294. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512838#comment-15512838
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/2533#discussion_r80010321
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
---
@@ -266,18 +265,20 @@ public void
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
- ObjectInputStream ois = new
ObjectInputStream(inView);
+ TypeSerializer namespaceSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+ TypeSerializer stateSerializer =
+
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
- TypeSerializer namespaceSerializer =
(TypeSerializer) ois.readObject();
- TypeSerializer stateSerializer =
(TypeSerializer) ois.readObject();
- StateTable stateTable = new
StateTable(stateSerializer,
+ StateTable stateTable = new StateTable(
+ stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
- for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+ for (int keyGroupIndex =
keyGroupRange.getStartKeyGroup(); keyGroupIndex <=
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --
I think i wanted to break this because it exceeds the line limit but then
decided against it because IntelliJ messed up the formatting for loops. Nothing
wrong there at all.
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512844#comment-15512844
]
Stephan Ewen commented on FLINK-4661:
-
What project is throwing that error? The test jar is generated in
{{flink-runtime}} in the usual way.
I see you are using a maven server, maybe that is the issue.
What other problem occur when compiling Flink? Most people compile it well with
default settings (including the flaky Travis CI service).
Could be a problem of your setup.
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
> --
>
> Key: FLINK-4661
> URL: https://issues.apache.org/jira/browse/FLINK-4661
> Project: Flink
> Issue Type: Bug
>Reporter: shijinkui
>
> [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could
> not resolve dependencies for project
> org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find
> org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in
> http://localhost:/repository/maven-public/ was cached in the local
> repository, resolution will not be reattempted until the update interval of
> nexus-releases has elapsed or updates are forced -> [Help 1]
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
> I can't find where this tests jar is generated.
> By the way, recently half month, I start to use flink. There is zero time I
> can compile the Flink project with default setting..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2533 @StephanEwen at least in the RocksDB backend we could remove user code completely. Right now, the only thing that needs to be serialized is the TypeSerializer from the ValueDescriptor. It is used in a check that users can not provide a descriptor with a different TypeSerializer than the one that was used initially. We might think about removing this to support versioning of TypeSerializers, but how can we somehow enforce compatibility between them? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512855#comment-15512855
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/2533
@StephanEwen at least in the RocksDB backend we could remove user code
completely. Right now, the only thing that needs to be serialized is the
TypeSerializer from the ValueDescriptor. It is used in a check that users can
not provide a descriptor with a different TypeSerializer than the one that was
used initially. We might think about removing this to support versioning of
TypeSerializers, but how can we somehow enforce compatibility between them?
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2170) Add OrcTableSource
[
https://issues.apache.org/jira/browse/FLINK-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512873#comment-15512873
]
Flavio Pompermaier commented on FLINK-2170:
---
Any news on this?
> Add OrcTableSource
> --
>
> Key: FLINK-2170
> URL: https://issues.apache.org/jira/browse/FLINK-2170
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Assignee: Owen O'Malley
>Priority: Minor
> Labels: starter
>
> Add a {{OrcTableSource}} to read data from an ORC file. The
> {{OrcTableSource}} should implement the {{ProjectableTableSource}}
> (FLINK-3848) and {{FilterableTableSource}} (FLINK-3849) interfaces.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4646) Add BipartiteGraph class
[ https://issues.apache.org/jira/browse/FLINK-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512880#comment-15512880 ] Vasia Kalavri commented on FLINK-4646: -- Hi [~StephanEwen], thanks! We've discussed this in the parent issue, yes. > Add BipartiteGraph class > > > Key: FLINK-4646 > URL: https://issues.apache.org/jira/browse/FLINK-4646 > Project: Flink > Issue Type: Sub-task > Components: Gelly >Reporter: Ivan Mushketyk > > Implement a class to represent a bipartite graph in Flink Gelly. Design > discussions can be found in the parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4646) Add BipartiteGraph class
[ https://issues.apache.org/jira/browse/FLINK-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512893#comment-15512893 ] Stephan Ewen commented on FLINK-4646: - Ah, I did not see that. Just thought that it would be good to check with some committers before embarking on a big implementation project. Thanks :-) > Add BipartiteGraph class > > > Key: FLINK-4646 > URL: https://issues.apache.org/jira/browse/FLINK-4646 > Project: Flink > Issue Type: Sub-task > Components: Gelly >Reporter: Ivan Mushketyk > > Implement a class to represent a bipartite graph in Flink Gelly. Design > discussions can be found in the parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3370) Add an aligned version of the window operator
[ https://issues.apache.org/jira/browse/FLINK-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512900#comment-15512900 ] Stephan Ewen commented on FLINK-3370: - Think of a "pane" as the unit of slide, sou add each element into a pane, and then compute the window as the union of multiple panes. It only makes sense for aligned windows (time, no custom non-time triggers). It is a bit tricky to implement in RocksDB, which is the main workhorse for large state right now. So we are delaying this until we have solved that problem... > Add an aligned version of the window operator > - > > Key: FLINK-3370 > URL: https://issues.apache.org/jira/browse/FLINK-3370 > Project: Flink > Issue Type: Improvement > Components: Windowing Operators >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > > The windowing operators currently follow a generic implementation for support > of unaligned windows. > We can gain efficiency by creating a variant that is optimized for aligned > windows: > - Aligned windows can use aligned triggers, which keep no per-key state > - Less trigger state means less checkpointing data > - Based on the aligned windows, we can create sliding event time windows > that do not replicate data into the different overlapping windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4544) TaskManager metrics are vulnerable to custom JMX bean installation
[
https://issues.apache.org/jira/browse/FLINK-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512915#comment-15512915
]
ASF GitHub Bot commented on FLINK-4544:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2445
Can we move this code out of the TaskManager as a whole, into a metrics
Utility?
We could make it reusable for the JobManager as well, by passing the metric
group where that should be added.
> TaskManager metrics are vulnerable to custom JMX bean installation
> --
>
> Key: FLINK-4544
> URL: https://issues.apache.org/jira/browse/FLINK-4544
> Project: Flink
> Issue Type: Bug
> Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.1.3
>
>
> The TaskManager's CPU load magic may fail when JMX providers are overwritten.
> The TaskManager logic checks if the class
> {{com.sun.management.OperatingSystemMXBean}} is available. If yes, it assumes
> that the {{ManagementFactory.getOperatingSystemMXBean()}} is of that type.
> That is not necessarily the case.
> This is visible in the Cassandra tests, as Cassandra overrides the JMX
> provider - every heartbeat causes an exception that is logged (See below),
> flooding the log, killing the heartbeat message.
> I would also suggest to move the entire metrics code out of the
> {{TaskManager}} class into a dedicated class {{TaskManagerJvmMetrics}}. That
> one can, with a static method, install the metrics into the TaskManager's
> metric group.
> Sample stack trace when default platform beans are overridden:
> {code}
> 23914 [flink-akka.actor.default-dispatcher-3] WARN
> org.apache.flink.runtime.taskmanager.TaskManager - Error retrieving CPU Load
> through OperatingSystemMXBean
> java.lang.IllegalArgumentException: object is not an instance of declaring
> class
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
> at scala.Option.map(Option.scala:145)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2351)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2348)
> at
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:32)
> at
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:20)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2444)
> at
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
> at
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:186)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3631)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3022)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.sendHeartbeatToJobManager(TaskManager.scala:1278)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:309)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
> at
> org.apache.flin
[GitHub] flink issue #2445: [FLINK-4544] Refactor old CPU metric initialization
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2445 Can we move this code out of the TaskManager as a whole, into a metrics Utility? We could make it reusable for the JobManager as well, by passing the metric group where that should be added. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512918#comment-15512918
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2533
I see, keeping the serializers for now makes probably sense.
It just seems that there are also user functions in there (like fold, etc)
- those should probably be removed. May mean that we have to inject them back
into the state descriptor later.
Orthogonal issue, so +1 for this change.
Merging this...
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2533 I see, keeping the serializers for now makes probably sense. It just seems that there are also user functions in there (like fold, etc) - those should probably be removed. May mean that we have to inject them back into the state descriptor later. Orthogonal issue, so +1 for this change. Merging this... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4580) Check that the RpcEndpoint supports the specified RpcGateway
[
https://issues.apache.org/jira/browse/FLINK-4580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512922#comment-15512922
]
ASF GitHub Bot commented on FLINK-4580:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2526
Nice!
+1 to merge this
> Check that the RpcEndpoint supports the specified RpcGateway
>
>
> Key: FLINK-4580
> URL: https://issues.apache.org/jira/browse/FLINK-4580
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> When calling {{RpcService.connect}} the user specifies the type of the
> {{RpcGateway}}. At the moment, it is not checked whether the {{RpcEndpoint}}
> actually supports the specified {{RpcGateway}}.
> I think it would be good to add a runtime check that the corresponding
> {{RpcEndpoint}} supports the specified {{RpcGateway}}. If not, then we can
> let the connect method fail fast.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2526: [FLINK-4580] [rpc] Report rpc invocation exceptions to th...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2526 Nice! +1 to merge this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2505: [FLINK-4628] provide user class loader during input split...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2505 Merging this... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment
[
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512926#comment-15512926
]
ASF GitHub Bot commented on FLINK-4628:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2505
Merging this...
> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the
> executing thread.
> In 2 we only have the system class loader available. If any classes need to
> be loaded then, we have a problem. Some InputFormats rely on code which
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the
> latest master.
> We should make the user class loader available when creating input splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2482: [FLINK-4579] [StateBackend] Add StateBackendFactory for R...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2482 Good point. I think reflect is good, if it does not add too much work. Keeping the modules separate is nice, I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2467: [FLINK-3719][web frontend] Moving the barrier between gra...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2467 The license is MIT, so that is fine. Can you update the LICENSE file with the dependency? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4579) Add StateBackendFactory for RocksDB Backend
[
https://issues.apache.org/jira/browse/FLINK-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512931#comment-15512931
]
ASF GitHub Bot commented on FLINK-4579:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2482
Good point. I think reflect is good, if it does not add too much work.
Keeping the modules separate is nice, I think.
> Add StateBackendFactory for RocksDB Backend
> ---
>
> Key: FLINK-4579
> URL: https://issues.apache.org/jira/browse/FLINK-4579
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: Aljoscha Krettek
>Assignee: Jark Wu
>
> Right now, we only have a {{StateBackendFactory}} for the {{FsStateBackend}}
> which means that users cannot specify to use the RocksDB backend in the flink
> configuration.
> If we add a factory for rocksdb we should also think about adding the rocksdb
> backend to the standard distribution lib, otherwise it is only usable if
> users manually place the rocks jars in the Flink lib folder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
Swapnil Chougule created FLINK-4663: --- Summary: Flink JDBCOutputFormat logs wrong WARN message Key: FLINK-4663 URL: https://issues.apache.org/jira/browse/FLINK-4663 Project: Flink Issue Type: Bug Components: Batch Connectors and Input/Output Formats Affects Versions: 1.1.2, 1.1.1 Environment: Across Platform Reporter: Swapnil Chougule Fix For: 1.1.3 Flink JDBCOutputFormat logs wrong WARN message as "Column SQL types array doesn't match arity of passed Row! Check the passed array..." even if there is no mismatch is SQL types array & arity of passed Row. (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats
[ https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512932#comment-15512932 ] ASF GitHub Bot commented on FLINK-3719: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2467 The license is MIT, so that is fine. Can you update the LICENSE file with the dependency? > WebInterface: Moving the barrier between graph and stats > > > Key: FLINK-3719 > URL: https://issues.apache.org/jira/browse/FLINK-3719 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: Niels Basjes >Assignee: Ivan Mushketyk > > It would be really useful if the separator between the graphical view of a > job topology at the top and the textual overview of the counters at the > bottom can be moved up/down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2467: [FLINK-3719][web frontend] Moving the barrier between gra...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2467 Piotr (who wrote most of the web ui) also wants to leave some comments. Let's wait for him. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats
[ https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512934#comment-15512934 ] ASF GitHub Bot commented on FLINK-3719: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2467 Piotr (who wrote most of the web ui) also wants to leave some comments. Let's wait for him. > WebInterface: Moving the barrier between graph and stats > > > Key: FLINK-3719 > URL: https://issues.apache.org/jira/browse/FLINK-3719 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: Niels Basjes >Assignee: Ivan Mushketyk > > It would be really useful if the separator between the graphical view of a > job topology at the top and the textual overview of the counters at the > bottom can be moved up/down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2458: [FLINK-4560] enforcer java version as 1.7
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2458 From my side, this is good to merge. Unless someone objects, I will merge this later today... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4560) enforcer java version as 1.7
[ https://issues.apache.org/jira/browse/FLINK-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512936#comment-15512936 ] ASF GitHub Bot commented on FLINK-4560: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2458 From my side, this is good to merge. Unless someone objects, I will merge this later today... > enforcer java version as 1.7 > > > Key: FLINK-4560 > URL: https://issues.apache.org/jira/browse/FLINK-4560 > Project: Flink > Issue Type: Improvement >Reporter: shijinkui > > 1. maven-enforcer-plugin add java version enforce > 2. maven-enforcer-plugin version upgrade to 1.4.1 > explicit require java version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2445: [FLINK-4544] Refactor old CPU metric initialization
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2445 that should be doable, yes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4544) TaskManager metrics are vulnerable to custom JMX bean installation
[
https://issues.apache.org/jira/browse/FLINK-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512941#comment-15512941
]
ASF GitHub Bot commented on FLINK-4544:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2445
that should be doable, yes.
> TaskManager metrics are vulnerable to custom JMX bean installation
> --
>
> Key: FLINK-4544
> URL: https://issues.apache.org/jira/browse/FLINK-4544
> Project: Flink
> Issue Type: Bug
> Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.1.3
>
>
> The TaskManager's CPU load magic may fail when JMX providers are overwritten.
> The TaskManager logic checks if the class
> {{com.sun.management.OperatingSystemMXBean}} is available. If yes, it assumes
> that the {{ManagementFactory.getOperatingSystemMXBean()}} is of that type.
> That is not necessarily the case.
> This is visible in the Cassandra tests, as Cassandra overrides the JMX
> provider - every heartbeat causes an exception that is logged (See below),
> flooding the log, killing the heartbeat message.
> I would also suggest to move the entire metrics code out of the
> {{TaskManager}} class into a dedicated class {{TaskManagerJvmMetrics}}. That
> one can, with a static method, install the metrics into the TaskManager's
> metric group.
> Sample stack trace when default platform beans are overridden:
> {code}
> 23914 [flink-akka.actor.default-dispatcher-3] WARN
> org.apache.flink.runtime.taskmanager.TaskManager - Error retrieving CPU Load
> through OperatingSystemMXBean
> java.lang.IllegalArgumentException: object is not an instance of declaring
> class
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
> at scala.Option.map(Option.scala:145)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2351)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2348)
> at
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:32)
> at
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:20)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2444)
> at
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
> at
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:186)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3631)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3022)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.sendHeartbeatToJobManager(TaskManager.scala:1278)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:309)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
> at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
> at
> org.apache.flink.runtime.testingUtils.TestingTaskManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingTaskManagerLike.scala:65)
> at scala.PartialFunction$OrElse.apply(Parti
[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Chougule updated FLINK-4663: Description: Flink JDBCOutputFormat logs wrong WARN message as "Column SQL types array doesn't match arity of passed Row! Check the passed array..." even if there is no mismatch is SQL types array & arity of passed Row. (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) It logs lot of unnecessary warning messages (one per row passed) in log files. was: Flink JDBCOutputFormat logs wrong WARN message as "Column SQL types array doesn't match arity of passed Row! Check the passed array..." even if there is no mismatch is SQL types array & arity of passed Row. (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512971#comment-15512971 ] Swapnil Chougule commented on FLINK-4663: - Hi Team, Can anybody give me permissions to assign same JIRA to me ? Thanks, Swapnil > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi updated FLINK-4663: -- Assignee: Swapnil Chougule > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule >Assignee: Swapnil Chougule > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-4662) Bump Calcite version up to 1.9
[ https://issues.apache.org/jira/browse/FLINK-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-4662: -- Assignee: Jark Wu > Bump Calcite version up to 1.9 > -- > > Key: FLINK-4662 > URL: https://issues.apache.org/jira/browse/FLINK-4662 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Jark Wu > > Calcite just released the 1.9 version. We should adopt it also in the Table > API especially for FLINK-4294. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2468: [FLINK-3580] [table] Add OVERLAPS function
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2468 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3580) Reintroduce Date/Time and implement scalar functions for it
[
https://issues.apache.org/jira/browse/FLINK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513099#comment-15513099
]
ASF GitHub Bot commented on FLINK-3580:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2468
> Reintroduce Date/Time and implement scalar functions for it
> ---
>
> Key: FLINK-3580
> URL: https://issues.apache.org/jira/browse/FLINK-3580
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> This task includes:
> {code}
> DATETIME_PLUS
> EXTRACT_DATE
> FLOOR
> CEIL
> CURRENT_TIME
> CURRENT_TIMESTAMP
> LOCALTIME
> LOCALTIMESTAMP
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 This needed another fix because in some tests we use the system class loader instead of a class loader instantiated by the BlobLibraryCacheManager. If we close that one, we cause tests to fail. The solution is to close only `FlinkUserCodeClassLoader`s. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 Merging after tests pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem
[
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513129#comment-15513129
]
ASF GitHub Bot commented on FLINK-4485:
---
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2499
This needed another fix because in some tests we use the system class
loader instead of a class loader instantiated by the BlobLibraryCacheManager.
If we close that one, we cause tests to fail. The solution is to close only
`FlinkUserCodeClassLoader`s.
> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this
> yarn session. It is the exact same job (java code) yet it gets different
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the
> parameters are about which data from which tables and the name of the target
> directory.
> After running several dozen jobs the jobs submission started to fail and we
> investigated.
> We found that the cause was that on the Yarn node which was hosting the
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID USER FD TYPE DEVICE SIZE
> NODE NAME
> java 15034 nbasjes 550r REG 253,17 66219695
> 245
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003
> (deleted)
> java 15034 nbasjes 551r REG 253,17 66219695
> 252
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007
> (deleted)
> java 15034 nbasjes 552r REG 253,17 66219695
> 267
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012
> (deleted)
> java 15034 nbasjes 553r REG 253,17 66219695
> 250
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005
> (deleted)
> java 15034 nbasjes 554r REG 253,17 66219695
> 288
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018
> (deleted)
> java 15034 nbasjes 555r REG 253,17 66219695
> 298
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025
> (deleted)
> java 15034 nbasjes 557r REG 253,17 66219695
> 254
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008
> (deleted)
> java 15034 nbasjes 558r REG 253,17 66219695
> 292
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019
> (deleted)
> java 15034 nbasjes 559r REG 253,17 66219695
> 275
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013
> (deleted)
> java 15034 nbasjes 560r REG 253,17 66219695
> 159
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002
> (deleted)
> java 15034 nbasjes 562r REG 253,17 66219695
> 238
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001
> (deleted)
> java 15034 nbasjes 568r REG 253,17 66219695
> 246
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004
> (deleted)
> java 15034 nbasjes 569r REG 253,17 66219695
> 255
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009
> (deleted)
> java 15034 nbasjes 571r REG 253,17 66219695
> 299
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026
> (deleted)
> java 15034 nbasjes 572r REG 253,17 66219695
> 293
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020
> (deleted)
> java 15034 nbasjes 574r REG 253,17 662196
[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem
[
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513131#comment-15513131
]
ASF GitHub Bot commented on FLINK-4485:
---
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2499
Merging after tests pass.
> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this
> yarn session. It is the exact same job (java code) yet it gets different
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the
> parameters are about which data from which tables and the name of the target
> directory.
> After running several dozen jobs the jobs submission started to fail and we
> investigated.
> We found that the cause was that on the Yarn node which was hosting the
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID USER FD TYPE DEVICE SIZE
> NODE NAME
> java 15034 nbasjes 550r REG 253,17 66219695
> 245
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003
> (deleted)
> java 15034 nbasjes 551r REG 253,17 66219695
> 252
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007
> (deleted)
> java 15034 nbasjes 552r REG 253,17 66219695
> 267
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012
> (deleted)
> java 15034 nbasjes 553r REG 253,17 66219695
> 250
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005
> (deleted)
> java 15034 nbasjes 554r REG 253,17 66219695
> 288
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018
> (deleted)
> java 15034 nbasjes 555r REG 253,17 66219695
> 298
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025
> (deleted)
> java 15034 nbasjes 557r REG 253,17 66219695
> 254
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008
> (deleted)
> java 15034 nbasjes 558r REG 253,17 66219695
> 292
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019
> (deleted)
> java 15034 nbasjes 559r REG 253,17 66219695
> 275
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013
> (deleted)
> java 15034 nbasjes 560r REG 253,17 66219695
> 159
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002
> (deleted)
> java 15034 nbasjes 562r REG 253,17 66219695
> 238
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001
> (deleted)
> java 15034 nbasjes 568r REG 253,17 66219695
> 246
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004
> (deleted)
> java 15034 nbasjes 569r REG 253,17 66219695
> 255
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009
> (deleted)
> java 15034 nbasjes 571r REG 253,17 66219695
> 299
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026
> (deleted)
> java 15034 nbasjes 572r REG 253,17 66219695
> 293
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020
> (deleted)
> java 15034 nbasjes 574r REG 253,17 66219695
> 256
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0010
> (deleted)
> java 15034 nbasjes 575r REG 253,17 66219695
> 302
> /tmp/blobStore-fbe9c4cf-1f85-48cb
[jira] [Commented] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
[ https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513157#comment-15513157 ] ASF GitHub Bot commented on FLINK-4496: --- Github user mxm commented on the issue: https://github.com/apache/flink/pull/2434 +1 Looks like a sensible change. Looking forward to fixing the `ContinuousFileReaderOperator`. > Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable. > > > Key: FLINK-4496 > URL: https://issues.apache.org/jira/browse/FLINK-4496 > Project: Flink > Issue Type: Sub-task >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2434: [FLINK-4496] Refactor the TimeServiceProvider to take a T...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2434 +1 Looks like a sensible change. Looking forward to fixing the `ContinuousFileReaderOperator`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
[ https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513162#comment-15513162 ] ASF GitHub Bot commented on FLINK-4496: --- Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2434 Thanks @mxm and @aljoscha ! I already have the followup on this open here: https://github.com/apache/flink/pull/2532 > Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable. > > > Key: FLINK-4496 > URL: https://issues.apache.org/jira/browse/FLINK-4496 > Project: Flink > Issue Type: Sub-task >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2434: [FLINK-4496] Refactor the TimeServiceProvider to take a T...
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2434 Thanks @mxm and @aljoscha ! I already have the followup on this open here: https://github.com/apache/flink/pull/2532 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2505: [FLINK-4628] provide user class loader during inpu...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2505 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment
[
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513198#comment-15513198
]
ASF GitHub Bot commented on FLINK-4628:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2505
> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the
> executing thread.
> In 2 we only have the system class loader available. If any classes need to
> be loaded then, we have a problem. Some InputFormats rely on code which
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the
> latest master.
> We should make the user class loader available when creating input splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2533 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513199#comment-15513199
]
ASF GitHub Bot commented on FLINK-4603:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2533
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4660) HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in a restarting loop
[
https://issues.apache.org/jira/browse/FLINK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513227#comment-15513227
]
Stephan Ewen commented on FLINK-4660:
-
We looked through this a bit, and the problem may be something else. Flink does
not close the {{FileSystem}} objects, but it also caches them, so there should
be only one {{FileSystem}} object per TaskManager.
The connections you see as open may be {{FsDataInputStream}} connections to S3,
reloading the state. Previous versions of Flink did not ensure that the streams
were closes in case that the recovery was intercepted by another failure (such
as File Not Found due to eventual consistency).
The latest version of Flink more thoroughly closes these streams. Can you check
if that fixes your problem?
For the eventual consistency issue, let's continue the discussion in
https://issues.apache.org/jira/browse/FLINK-4218
> HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in
> a restarting loop
> ---
>
> Key: FLINK-4660
> URL: https://issues.apache.org/jira/browse/FLINK-4660
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Zhenzhong Xu
>Priority: Critical
> Attachments: Screen Shot 2016-09-20 at 2.49.14 PM.png, Screen Shot
> 2016-09-20 at 2.49.32 PM.png
>
>
> Flink job with checkpoints enabled and configured to use S3A file system
> backend, sometimes experiences checkpointing failure due to S3 consistency
> issue. This behavior is also reported by other people and documented in
> https://issues.apache.org/jira/browse/FLINK-4218.
> This problem gets magnified by current HadoopFileSystem implementation, which
> can potentially leak S3 client connections, and eventually get into a
> restarting loop with “Timeout waiting for a connection from pool” exception
> thrown from aws client.
> I looked at the code, seems HadoopFileSystem.java never invoke close() method
> on fs object upon failure, but the FileSystem may be re-initialized every
> time the job gets restarted.
> A few evidence I observed:
> 1. When I set the connection pool limit to 128, and below commands shows 128
> connections are stuck in CLOSE_WAIT state.
> !Screen Shot 2016-09-20 at 2.49.14 PM.png|align=left, vspace=5!
> 2. task manager logs indicates that state backend file system consistently
> getting initialized upon job restarting.
> !Screen Shot 2016-09-20 at 2.49.32 PM.png!
> 3. Log indicates there is NPE during cleanning up of stream task which was
> caused by “Timeout waiting for connection from pool” exception when trying to
> create a directory in S3 bucket.
> 2016-09-02 08:17:50,886 ERROR
> org.apache.flink.streaming.runtime.tasks.StreamTask - Error during cleanup of
> stream task
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.cleanup(OneInputStreamTask.java:73)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
> at java.lang.Thread.run(Thread.java:745)
> 4.It appears StreamTask from invoking checkpointing operation, to handling
> failure, there is no logic associated with closing Hadoop File System object
> (which internally includes S3 aws client object), which resides in
> HadoopFileSystem.java.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting
[
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513234#comment-15513234
]
Stephan Ewen commented on FLINK-4218:
-
[~zhenzhongxu] Adding you to this conversation.
My assumptions is that the problem is as follows: The file exists and is
consistently visible (if I understand S3 correctly), but the parent directory's
metadata is eventual consistent. The operation that fails here is the lookup of
the file size, which is in most file systems an operation on the parent
directory, not the file itself. So that would explain why it occasionally fails.
What is the best way to fix this? Simply have a few retries? If it still fails
after the retries, simply use a special value for unknown file size?
The state size information is used mainly for informational purposes, like in
the web UI and in metrics.
> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..."
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of
> receiving checkpoint barrier
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory:
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
> at
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
> at
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
> at
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
> ... 8 more
> {code}
> File actually exists on S3.
> I suppose it is related to some race conditions with S3 but would be good to
> retry a few times before stop task execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513241#comment-15513241
]
ASF GitHub Bot commented on FLINK-4663:
---
GitHub user swapnil-chougule opened a pull request:
https://github.com/apache/flink/pull/2534
[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed
Fixed wrong WARN message logged by JDBCOutputFormat while adding row
(writing record to prepared statement) as
"Column SQL types array doesn't match arity of passed Row! Check the passed
array..."
even if there is no mismatch is SQL types array & arity of passed Row.
[FLINK-4663]
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-4663]")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- No need to change documentation for same.
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/swapnil-chougule/flink FLINK-4663
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2534.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2534
commit a3a3a76164c717f079887a641598e3b9aa6ce484
Author: swapnil-chougule
Date: 2016-09-22T12:31:20Z
[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed
> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
> Issue Type: Bug
> Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as
> "Column SQL types array doesn't match arity of passed Row! Check the passed
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2534: [FLINK-4663] Flink JDBCOutputFormat logs wrong WAR...
GitHub user swapnil-chougule opened a pull request:
https://github.com/apache/flink/pull/2534
[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed
Fixed wrong WARN message logged by JDBCOutputFormat while adding row
(writing record to prepared statement) as
"Column SQL types array doesn't match arity of passed Row! Check the passed
array..."
even if there is no mismatch is SQL types array & arity of passed Row.
[FLINK-4663]
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-4663]")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- No need to change documentation for same.
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/swapnil-chougule/flink FLINK-4663
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2534.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2534
commit a3a3a76164c717f079887a641598e3b9aa6ce484
Author: swapnil-chougule
Date: 2016-09-22T12:31:20Z
[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Chougule updated FLINK-4663: Priority: Minor (was: Major) > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule >Assignee: Swapnil Chougule >Priority: Minor > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Chougule resolved FLINK-4663. - Resolution: Fixed Resolved. Created PR https://github.com/apache/flink/pull/2534 Kindly review. Regards, Swapnil > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule >Assignee: Swapnil Chougule >Priority: Minor > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reopened FLINK-4663: -- Hi [~the.swapni...@gmail.com], thanks for providing a fix! We will close the issue once the pull request is merged. Thanks, Fabian > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule >Assignee: Swapnil Chougule >Priority: Minor > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2534: [FLINK-4663] Flink JDBCOutputFormat logs wrong WARN messa...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2534 +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message
[ https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513265#comment-15513265 ] ASF GitHub Bot commented on FLINK-4663: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/2534 +1 to merge. > Flink JDBCOutputFormat logs wrong WARN message > -- > > Key: FLINK-4663 > URL: https://issues.apache.org/jira/browse/FLINK-4663 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.1.1, 1.1.2 > Environment: Across Platform >Reporter: Swapnil Chougule >Assignee: Swapnil Chougule >Priority: Minor > Fix For: 1.1.3 > > > Flink JDBCOutputFormat logs wrong WARN message as > "Column SQL types array doesn't match arity of passed Row! Check the passed > array..." > even if there is no mismatch is SQL types array & arity of passed Row. > > (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java) > It logs lot of unnecessary warning messages (one per row passed) in log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly
[
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513302#comment-15513302
]
Timo Walther commented on FLINK-4263:
-
I also had a look at it. Replacing {{Seq[Row]}} field by {{Seq[List]}} does
only solve the current problem. What happens if we have a row of rows or row of
POJOs. I think maybe we should also code generate the values input format.
Otherwise we also have to make sure that the contents of the values are always
serializable no matter which data types may be added in future. [~jark] do you
wanna still fix this issue? I could also assign it to me.
> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data
> Source "at translateToPlan(DataSetValues.scala:88)
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code
> wrapper class
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
> java.io.NotSerializableException: org.apache.flink.api.table.Row
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
> at
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
> at
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
> at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
> at
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by:
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could
> not write the user code wrapper class
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
> java.io.NotSerializableException: org.apache.flink.api.table.Row
> at
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
> ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>
[jira] [Commented] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513311#comment-15513311
]
Timo Walther commented on FLINK-4661:
-
Today [~fhueske] and me also have problems in compiling {{flink-table}}. Maybe
it is just a Maven central issue. Or have we changed Maven dependencies
recently?
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
> --
>
> Key: FLINK-4661
> URL: https://issues.apache.org/jira/browse/FLINK-4661
> Project: Flink
> Issue Type: Bug
>Reporter: shijinkui
>
> [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could
> not resolve dependencies for project
> org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find
> org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in
> http://localhost:/repository/maven-public/ was cached in the local
> repository, resolution will not be reattempted until the update interval of
> nexus-releases has elapsed or updates are forced -> [Help 1]
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
> I can't find where this tests jar is generated.
> By the way, recently half month, I start to use flink. There is zero time I
> can compile the Flink project with default setting..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4554) Add support for array types
[
https://issues.apache.org/jira/browse/FLINK-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther reassigned FLINK-4554:
---
Assignee: Timo Walther
> Add support for array types
> ---
>
> Key: FLINK-4554
> URL: https://issues.apache.org/jira/browse/FLINK-4554
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Support creating arrays:
> {code}ARRAY[1, 2, 3]{code}
> Access array values:
> {code}myArray[3]{code}
> And operations like:
> {{UNNEST, UNNEST WITH ORDINALITY, CARDINALITY}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4556) Make Queryable State Key-Group Aware
[
https://issues.apache.org/jira/browse/FLINK-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513330#comment-15513330
]
ASF GitHub Bot commented on FLINK-4556:
---
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/2523
please review @aljoscha or @StephanEwen
> Make Queryable State Key-Group Aware
>
>
> Key: FLINK-4556
> URL: https://issues.apache.org/jira/browse/FLINK-4556
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.2.0
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
>Priority: Blocker
>
> The recent introduction of key-grouped state breaks queryable state because
> the JobManager does not yet forward the client to the correct TaskManager
> based on key-group ranges.
> This will either have to be implemented on the JobManager side, i.e. in
> {{AkkaKvStateLocationLookupService}} or on the {{TaskManager}} when state is
> registered. The JobManager can know the mapping because it should know the
> {{parallelism}}/{{maxParallelism}} which it can use to determine where the
> state for a key-group is stored. The {{TaskManager}} send a
> {{NotifyKvStateRegistered}} message that already contains a {{keyGroupIndex}}
> field that is not useful/correct at the moment, though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2523: [FLINK-4556] Make Queryable State Key-Group Aware
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2523 please review @aljoscha or @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4587) Yet another java.lang.NoSuchFieldError: INSTANCE
[
https://issues.apache.org/jira/browse/FLINK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513337#comment-15513337
]
Stephan Ewen commented on FLINK-4587:
-
It depends on which maven version you are using.
Maven 3.0.x woulds directly
If you use Maven 3.3.x, you need to rebuild "flink-dist". It is a problem of
later versions of Maven and the Maven Shade Plugin
{code}
mvn clean package -DskipTests
cd flink-dist
mvn clean package -DskipTests
{code}
> Yet another java.lang.NoSuchFieldError: INSTANCE
>
>
> Key: FLINK-4587
> URL: https://issues.apache.org/jira/browse/FLINK-4587
> Project: Flink
> Issue Type: Bug
> Components: Build System, Local Runtime
>Affects Versions: 1.2.0
> Environment: Latest SNAPSHOT
>Reporter: Renkai Ge
> Attachments: diff in mvn clean package.png, flink-explore-src.zip
>
>
> For some reason I need to use org.apache.httpcomponents:httpasyncclient:4.1.2
> in flink.
> The source file is:
> {code}
> import org.apache.flink.streaming.api.scala._
> import org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory
> /**
> * Created by renkai on 16/9/7.
> */
> object Main {
> def main(args: Array[String]): Unit = {
> val instance = ManagedNHttpClientConnectionFactory.INSTANCE
> println("instance = " + instance)
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromCollection(1 to 100)
> val result = stream.map { x =>
> x * 2
> }
> result.print()
> env.execute("xixi")
> }
> }
> {code}
> and
> {code}
> name := "flink-explore"
> version := "1.0"
> scalaVersion := "2.11.8"
> crossPaths := false
> libraryDependencies ++= Seq(
> "org.apache.flink" %% "flink-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-connector-kafka-0.8" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-streaming-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-clients" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.httpcomponents" % "httpasyncclient" % "4.1.2"
> )
> {code}
> I use `sbt assembly` to get a fat jar.
> If I run the command
> {code}
> java -cp flink-explore-assembly-1.0.jar Main
> {code}
> I get the result
> {code}
> instance =
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory@4909b8da
> log4j:WARN No appenders could be found for logger
> (org.apache.flink.api.scala.ClosureCleaner$).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Connected to JobManager at Actor[akka://flink/user/jobmanager_1#-1177584915]
> 09/07/2016 12:05:26 Job execution switched to status RUNNING.
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to SCHEDULED
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to DEPLOYING
> ...
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(20/24) switched to RUNNING
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(19/24) switched to RUNNING
> 15> 30
> 20> 184
> ...
> 19> 182
> 1> 194
> 8> 160
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to FINISHED
> ...
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(1/24) switched to FINISHED
> 09/07/2016 12:05:26 Job execution switched to status FINISHED.
> {code}
> Nothing special.
> But if I run the jar by
> {code}
> ./bin/flink run shop-monitor-flink-assembly-1.0.jar
> {code}
> I will get an error
> {code}
> $ ./bin/flink run flink-explore-assembly-1.0.jar
> Cluster configuration: Standalone cluster with JobManager at /127.0.0.1:6123
> Using address 127.0.0.1:6123 to connect to JobManager.
> JobManager web interface address http://127.0.0.1:8081
> Starting execution of program
>
> The program finished with the following exception:
> java.lang.NoSuchFieldError: INSTANCE
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:53)
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:57)
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:47)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:75)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:83)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:64)
> at Main$.main
[jira] [Closed] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-4603.
---
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-4603) KeyedStateBackend cannot restore user code classes
[
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-4603.
-
Resolution: Fixed
Fixed via 3b8fe95ec728d59e3ffba2901450c56d7cca2b24
> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code
> classes. I suspect that we don't use the user code class loader to
> deserialize the state.
> The solution seems to be to forward the user code class loader to the
> {{KeyedStateBackends}} when restoring state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4628) User class loader unavailable during input split assignment
[
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-4628.
---
> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.2.0
>
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the
> executing thread.
> In 2 we only have the system class loader available. If any classes need to
> be loaded then, we have a problem. Some InputFormats rely on code which
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the
> latest master.
> We should make the user class loader available when creating input splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-4628) User class loader unavailable during input split assignment
[
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-4628.
-
Resolution: Fixed
Fix Version/s: 1.2.0
Fixed via 345b2529a8acdd59d67e89ea930ec69ad69a55d3
> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.2.0
>
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the
> executing thread.
> In 2 we only have the system class loader available. If any classes need to
> be loaded then, we have a problem. Some InputFormats rely on code which
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the
> latest master.
> We should make the user class loader available when creating input splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly
[
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513401#comment-15513401
]
Jark Wu commented on FLINK-4263:
It seems that the data types in {{VALUES}} are always RexLiteral which should
be basic types and could be serializable ?
> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data
> Source "at translateToPlan(DataSetValues.scala:88)
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code
> wrapper class
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
> java.io.NotSerializableException: org.apache.flink.api.table.Row
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
> at
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
> at
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
> at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
> at
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by:
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could
> not write the user code wrapper class
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
> java.io.NotSerializableException: org.apache.flink.api.table.Row
> at
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
> at
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
> ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.jav
