Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/5123
@tillrohrmann sorry for the late update. you are right, I have change the
pr to just removing the temporary file after uploaded to hdfs.
---
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/5122
---
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/5122
@StephanEwen Thanks for reviewing, I have update the pr according to your
comment.
---
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/5123
@aljoscha yes, the task manager config file is created in $PWD dir, and
files created in temporary folder will be cleared automatically by junit.
---
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/5123
[FLINK-8201] YarnResourceManagerTest causes license checking failure
YarnResourceManagerTest generates a temporary taskmanager config file in
flink-yarn module
root folder and never clear
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/5122
[FLINK-8200] RocksDBAsyncSnapshotTest should use temp fold instead of fold
with fixed name
Using fixed folder in test makes test fail when different users run the
test in the same machine. We
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3716
[FLINK-6298]Local execution is not setting RuntimeContext for
RichOutputFormat
call set RuntimeContext OutputFormat when the OutputFormat is
RichOutputFormat
You can merge this pull request
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3709#discussion_r111308326
--- Diff:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/ExecutionGraphHolder.java
---
@@ -48,7 +52,41 @@
private
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
hi, @greghogan , do you have any new comment about the changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
@greghogan I come across this bug when we are trying to write some objects
to a byte array: at first, I wrote a 0 as initial size and then the serialized
object, after that, I seek back
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
@greghogan Currently we check the position with the EndPosition which is
the size of buffer instead of a max written size , so it is still possible to
leave holes when keeping unmodified
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
@greghogan Thanks for the review. I have add non-negative check and update
the test with ExpectedException. But I don't think we need to call toString in
the test since we have check the position
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
hi, @StefanRRichter , I have add check in setPosition in
ByteArrayInputSteamWithPos, there is no dependency on this method too, so I
rename the method from setPos to setPosition for unifying
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
@greghogan comments addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3599
Hi, I may have described my concern wrongly in the last comment, my concern
is that in yarn it is possible that two application master running at the same
time:
eg: RM launches a AM
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3599
Hi, @WangTaoTheTonic I think we can improve the reaction of
ZookeeperLeaderElectionService on zookeeper connection expired or other errors
instead of introducing the AlwaysLeaderService
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3595
@greghogan thanks for the review, I have changed the change test to use
explicit buffer size.
you are right that currently there is no dependency on this method. We used
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3595
[FLINK-6162]Fix bug in ByteArrayOutputStreamWithPos#setPosition
Currently the precondition check in setPosition will fail when the buffer
is full:
{{Preconditions.checkArgument(position
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3467#discussion_r105081191
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
---
@@ -265,11 +281,15 @@ public String toString
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3438
thanks for explaination, I have such concern because we have just suggested
our users to use processFunction to implement their jobs, they need to change
their code too when we sync
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3438#discussion_r104334699
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/ProcessFunction.java
---
@@ -19,30 +19,35 @@
package
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3388
@tillrohrmann sorry for late response, I have addressed the comments.
Thinks for the review.
Currently for standalone, and mesos, there are already some ways to do
things like
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878983
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnCLI.java ---
@@ -85,6 +89,10 @@ public FlinkYarnCLI(String shortPrefix, String
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878445
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878609
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -131,6 +134,8 @@
/** Lazily initialized list
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878642
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -248,6 +253,14 @@ public void addShipFiles(List
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878685
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -742,6 +755,24 @@ public FileVisitResult
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878413
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878546
--- Diff:
flink-fs-tests/src/test/java/org/apache/flink/hdfstests/FileUtilsTest.java ---
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878668
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -742,6 +755,24 @@ public FileVisitResult
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878523
--- Diff:
flink-fs-tests/src/test/java/org/apache/flink/hdfstests/FileUtilsTest.java ---
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878488
--- Diff:
flink-fs-tests/src/test/java/org/apache/flink/hdfstests/FileUtilsTest.java ---
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878472
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878375
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878350
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878339
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878313
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -258,7 +261,82 @@ public static boolean deletePathIfEmpty(FileSystem
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3334
Currently the `numUnsuccessfulCheckpointsTriggers` will be reset after a
successful trigger instead of a successful checkpoint. But I think it is rare
actually for triggering failure
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3190
Can we just use the `${project.build.directory}` as `java.io.tmpdir` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3388
[FLINK-5815] Add resource files configuration for Yarn Mode
This PR add three common resource configuration options to yarn mode, which
allow user to set single file resource from both local
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3336#discussion_r102381085
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/state/MapStateDescriptor.java
---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3371
hi @tillrohrmann, the CI is failing because of my changes, could you help
take a look it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3371
[FLINK-5836] Fix race condition between offer slot and submit task
The solution is the same as what till described in jira: activating the
slots when reserving them on `TaskExecutor` before
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3288
hi @StephanEwen , I have move the changes to child module pom,
`flink-tests` and `flink-hadoop-compatibility` need to be changed too, because
the mapreduce related cases will load output
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3341
[FLINK-5817]Fix test concurrent execution failure by test dir conflicts.
use TemporaryFold to create temp files and folds for UT
You can merge this pull request into a Git repository by running
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3288
ok, I will move the changes to filesystem-connector and fs-tests poms.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/3288
hi @StephanEwen, the cases in flink-filesystem-connector will fail first in
our building machine. I have go through the cases in flink-fs-tests, I think
they will fail too if I remove the change
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/3288
[FLINK-5749]unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build
machine failing the UT and IT
Currently when we are trying to build flink on a machine with HADOOP_HOME
environment
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2356
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
all of the test in travis have passed. thanks for merging this. ^_^
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
I have commented above, it is to make sure that the customized
configuration work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
@aljoscha i have rebase the newest master branch, and forward port the
changes in rollingsink to bucketsink
---
If your project is set up for it, you can reply to this email and have your
reply
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
sorry,@aljoscha, I have miss the last message in august, I will update the
pr quite soon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2375
discarding this pr, since we define the rpc interfaces in specific subtasks
such as task executor registration, etc.
---
If your project is set up for it, you can reply to this email and have
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2375
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2401
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user wenlong88 reopened a pull request:
https://github.com/apache/flink/pull/2401
[FLINK-4443][rpc] Add support in RpcCompletenessTest for inheritance of
RpcGateway and RpcEndpoint
RpcCompletenessTest needs to support RpcGateway which is composited by some
basic functions
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2401
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2401
@tillrohrmann all comments addressed. I have removed the `NativeMethod`
annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/2401
[FLInk-4443][rpc] Add support in RpcCompletenessTest for inheritance of
RpcGateway and RpcEndpoint
adding support for RpcCompletenessTest needs to support RpcGateway which is
composited by some
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2381
ok, I will open a issue for change RpcCompletenessTest to support
inheritance of RpcGateway and RpcEndpoint later.
---
If your project is set up for it, you can reply to this email and have your
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2381
@tillrohrmann I agrees that `getAddress` in gateway can provide some
facilities, I think that we can treat a special method `getAddress` executed
locally just like the timeout argument, which
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2381
Coming back to the initial motivation of this PR, It is because that I
tried to get the address from the RpcService, and the RpcService provides a api
`getAddress(RpcGateway gateway)` but can
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2381
Hi, @tillrohrmann
1. It looks good if you think it is not a bit deal that the interface
RpcGateway contains functions defined for specific business logic and functions
defined for frameworks
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2381
If there is a `RpcGateway.getAddress`, it will be more friendly in api.
But I also think mixing the user defined RPC methods and framework interfaces
will be a bit of odd and needs specific
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/2381#discussion_r75157339
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcService.java
---
@@ -197,11 +201,11 @@ public void stopService
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/2381
[FLINK-4414][cluster management] Remove restriction on RpcService.getAddress
currently RpcService provide only address of the endpoint, I think rpc
service serve both the endpoint create
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/2375
[FLINK-4375][cluster management]Introduce rpc protocols implemented by job
manager
1. Add basic RPC interface provided by job master including
2. improve RpcCompletenessTest to support
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen you are right. But in specific situation, we may need some
temporary compromise do make the system work well, and then remove the
compromised points latter as soon as possible
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
the change is a little bigger than expected. because {{FileSystem}} will be
cached once created ,to make sure the user defined configuration work,
{{fs.%s.impl.disable.cache}} need to be set
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/2353#discussion_r74533793
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/taskexecutor/TaskExecutorToResourceManagerConnection.java
---
@@ -0,0 +1,80
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/2356
[FLINK-4378]Enable RollingSink to custom HDFS client configuration
Adding a new interface to rolling sink
You can merge this pull request into a Git repository by running:
$ git pull https
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
3q, @aljoscha
Maybe we can use different db instance for different key group. this
approach can only work well when key group number is limited to a certain
number which can not be too large
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@aljoscha I am curious at what problem and incompatibility between
semi-aync snapshot and key-group, can you explain some background information?
---
If your project is set up for it, you can
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen
http://rocksdb.org/blog/2609/use-checkpoints-for-efficient-snapshots/
since sst files is immutable once created, in when doing checkpoint,
rocksdb creates hard link for all live
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@aljoscha we use rocksdb checkpoint machanism to do the semi-async
checkpoint, which use hard link to make checkpoint, cost quite a little IO and
time in synchronized phrase. This works well even
77 matches
Mail list logo
