There is a terasort implementation with deprecated API.
https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/recordJobs/sort/TeraSort.java
AFAIK, there is no implementation with current API.
Regards,
Chiwan Park
On Jun 4, 2015, at 12:17 AM, Bill Sparks
Hi. You can build Flink with Scala 2.11 with scala-2.11 profile in master
branch.
`mvn clean install -DskipTests -P \!scala-2.10,scala-2.11` command builds Flink
with Scala 2.11.
Regards,
Chiwan Park
On Jun 10, 2015, at 9:56 PM, Flavio Pompermaier pomperma...@okkam.it wrote:
Nice!
On 10
No. Currently, there are no Flink binaries with scala 2.11 which are
downloadable.
Regards,
Chiwan Park
On Jun 10, 2015, at 10:18 PM, Philipp Goetze philipp.goe...@tu-ilmenau.de
wrote:
Thank you Chiwan!
I did not know the master has a 2.11 profile.
But there is no pre-built Flink
But I think uploading Flink API with scala 2.11 to maven repository is nice
idea.
Could you create a JIRA issue?
Regards,
Chiwan Park
On Jun 10, 2015, at 10:23 PM, Chiwan Park chiwanp...@icloud.com wrote:
No. Currently, there are no Flink binaries with scala 2.11 which are
downloadable
Hi.
Flink is not DBMS. There is no equivalent operation of insert, update, remove.
But you can use map[1] or filter[2] operation to create modified dataset.
I recommend you some sildes[3][4] to understand Flink concepts.
Regards,
Chiwan Park
[1]
http://ci.apache.org/projects/flink/flink-docs
It represents the folder containing the hadoop config files. :)
Regards,
Chiwan Park
On Jun 25, 2015, at 10:07 PM, Flavio Pompermaier pomperma...@okkam.it wrote:
fs.hdfs.hadoopconf represents the folder containing the hadoop config files
(*-site.xml) or just one specific hadoop config
-docs-master/internals/logging.html
[3] http://stackoverflow.com/a/3810936
Regards,
Chiwan Park
On Jun 19, 2015, at 8:05 PM, Juan Fumero
juan.jose.fumero.alfo...@oracle.com wrote:
Hi,
is there any option (from API level) to redirect the log messages to a
file instead of stdout
I’m interested in working on this. :) I’ll assign to me.
Regards,
Chiwan Park
On Jun 21, 2015, at 8:22 AM, Robert Metzger rmetz...@apache.org wrote:
Okay, it seems like we have consensus on this. Who is interested in working
on this? https://issues.apache.org/jira/browse/FLINK-2200
://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2Ftasks%2Ftasks-java-local-configuration.htm
[2]
https://www.jetbrains.com/idea/help/creating-and-editing-run-debug-configurations.html
Regards,
Chiwan Park
On Jun 17, 2015, at 2:01 PM, Sebastian s...@apache.org wrote:
Hi,
Flink has
, …, etc. with
version variation.
So we can reduce a number of deployed modules.
Regards,
Chiwan Park
On Jun 13, 2015, at 9:17 AM, Robert Metzger rmetz...@apache.org wrote:
I agree that we should ship a 2.11 build of Flink if downstream projects need
that.
The only thing that we should
Note that sortPartition is implemented in 0.9. Following link shows the example
of sortPartition.
http://ci.apache.org/projects/flink/flink-docs-master/apis/dataset_transformations.html#sort-partition
Regards,
Chiwan Park
On Jun 2, 2015, at 5:51 PM, hagersaleh loveallah1...@yahoo.com wrote
.
DataSetCustomer sorted = customers.groupBy(2).sortGroup(0,
Order.DESCENDING).first(10);
System.out.println(sorted.print());
Note that Flink does not support global sort (FLINK-598) but only support local
sort currently. The sortGroup API means that sorting for each group.
Regards,
Chiwan Park
On Jun
@Alexander I’m happy to hear that you want to help me. If you help me, I really
appreciate. :)
Regards,
Chiwan Park
On Jul 2, 2015, at 2:57 PM, Alexander Alexandrov
alexander.s.alexand...@gmail.com wrote:
@Chiwan: let me know if you need hands-on support. I'll be more then happy
I found that the patch had been merged to upstream. [1] :)
Regards,
Chiwan Park
[1] https://github.com/apache/flink/pull/855
On Jul 3, 2015, at 5:26 PM, Welly Tambunan if05...@gmail.com wrote:
Thanks Chiwan,
Glad to hear that.
Cheers
On Fri, Jul 3, 2015 at 3:24 PM, Chiwan
Hi Hermann,
In 16 page of Slim’s slides [1], there is a pre-installed virtual machine based
on VMWare. I haven’t run Flink on that machine. But maybe It works.
Regards,
Chiwan Park
[1]
http://www.slideshare.net/sbaltagi/apache-flinkcrashcoursebyslimbaltagiandsrinipalthepu
On Aug 19, 2015
Hi,
We already know this issue. There are some problems in Apache Infrastructure.
Infra Team is working on this issue. You can see progress via a blog post [1].
It will be okay soon.
Regards,
Chiwan Park
[1] https://blogs.apache.org/infra/entry/buildbot_master_currently_off_line
On Jun 30
If you would search a graphical interface for data analytics like Jupyter, you
should look Apache Zeppelin [1].
Apache Zeppelin is a web-based notebook. It supports Scala, Spark and Flink.
Regards,
Chiwan Park
[1] https://zeppelin.incubator.apache.org
On Jul 13, 2015, at 9:23 PM, Till
Hi, If you use `partitionCustom()` method [1] with custom partitioner, you can
guarantee the order of partition.
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/java/DataSet.html#partitionCustom
You can increase Flink managed memory by increasing Taskmanager JVM Heap
(taskmanager.heap.mb) in flink-conf.yaml.
There is some explanation of options in Flink documentation [1].
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#common-options
documentation. The documentation will help you to understand the
structure of Flink program.
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html#data-sinks
[2]
https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis
. If you want to know more detail of key specifying
method
in Flink, please see the documentation [2] in Flink homepage.
Regards,
Chiwan Park
[1] https://gist.github.com/chiwanpark/e71d27cc8edae8bc7298
[2]
https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#specifying
the functions.
I think that It is good to read Batch API section of Flink documentation for
you.
If you have a question for the example, please reply mail to user mailing list.
Regards,
Chiwan Park
[1] https://gist.github.com/chiwanpark/5e2a6ac00b7e0bf85444
[2]
https://ci.apache.org/projects
Hi, you should use RichMapFunction not MapFunction. The difference between
RichMapFunction and MapFunction is described in Flink documentation [1].
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#rich-functions
On Jul 12, 2015, at 7
Because there is no default implementations like forany in scala, I use forall
method. Note that ANY (condition) is equivalent as NOT ALL (NOT condition).
Regards,
Chiwan Park
On Jul 12, 2015, at 5:39 AM, hagersaleh loveallah1...@yahoo.com wrote:
why in this use ! and = in handle Any
Hi Michele,
We’re doing release process for 0.9.1. Ufuk Celebi will start vote for 0.9.1
release soon.
Regards,
Chiwan Park
On Aug 27, 2015, at 6:49 PM, Michele Bertoni
michele1.bert...@mail.polimi.it wrote:
Hi everybody,
I am still waiting for version 0.9.1 to solve this problem, any
Additionally, If you have any questions about contributing, please send a mail
to dev mailing list.
Regards,
Chiwan Park
On Aug 27, 2015, at 2:11 PM, Chiwan Park chiwanp...@apache.org wrote:
Hi Naveen,
There is a guide document [1] about contribution in homepage. Please read
first
`, or `easyfix`.
Happy contributing!
Regards,
Chiwan Park
[1] http://flink.apache.org/how-to-contribute.html
[2] http://flink.apache.org/coding-guidelines.html
[3]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20starter
you want to use broadcast variable. You
can do same thing with filter and join operations. Here is my implementation
[1].
Regards,
Chiwan Park
[1] https://gist.github.com/chiwanpark/a0b0269c9a9b058d15d3
> On Sep 4, 2015, at 3:51 AM, hagersaleh <loveallah1...@yahoo.com> wrote:
>
+1 for dropping Hadoop 2.2.0
Regards,
Chiwan Park
> On Sep 4, 2015, at 5:58 AM, Ufuk Celebi <u...@apache.org> wrote:
>
> +1 to what Robert said.
>
> On Thursday, September 3, 2015, Robert Metzger <rmetz...@apache.org> wrote:
> I think most cloud providers moved b
Hi Giacomo,
You should set your field as public. If you are set your field as private or
protected, the class must provide getter and setter to be treated as POJO.
Maybe the documentation in homepage [1] would be helpful.
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink
[3] https://github.com/apache/flink/pull/1134
Regards,
Chiwan Park
> On Sep 17, 2015, at 1:33 AM, Chiwan Park <chiwanp...@apache.org> wrote:
>
> It seems like a bug of CsvInputFormat. I succeed in reproducing in my local
> machine.
> I will create a JIRA issue for this and
It seems like a bug of CsvInputFormat. I succeed in reproducing in my local
machine.
I will create a JIRA issue for this and submit a patch to fix it.
Which version of Flink are used?
Regards,
Chiwan Park
> On Sep 17, 2015, at 12:20 AM, Giacomo Licari <giacomo.lic...@gmail.com> wrote:
Hi Felix,
You can change the listening port of jobmanager web frontend by setting
`jobmanager.web.port` in configuration (conf/flink-conf.yml).
I attached a link of documentation [1] about this.
Regards,
Chiwan Park
[1]
https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup
Hi Deng,
The jarFiles parameter of `createRemoteEnvironment` means that the path of your
custom library jar. If you don’t need custom library, you can omit the
parameter.
Regards,
Chiwan Park
> On Sep 25, 2015, at 10:48 AM, Deng Jie <smartdeng...@gmail.com> wrote:
>
> Dear F
Hi Deng,
It sounds weird. In code [1], `jarFiles` parameter is defined as a varargs
parameter. From this, we can omit the parameter.
Which version of Flink are you using?
Regards,
Chiwan Park
[1]
https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java
Oh, sorry for wrong information.
I have misunderstood about `jarFiles` parameter.
Regards,
Chiwan Park
> On Sep 25, 2015, at 5:27 PM, Fabian Hueske <fhue...@gmail.com> wrote:
>
> Hi Deng Jie,
>
> your Flink program needs to be packaged into a JAR file.
> The Flink qui
because of lack of permission. How can I solve this problem?
Regards,
Chiwan Park
t;
>
> --
> Tzu-Li Tai (Gordon Tai)
> 戴資力
>
> National Cheng Kung University, Graduate Institute of Computer and
> Communication Engineering
> High Performance Parallel and Distributed Systems Laboratory (HPDS Lab)
> 國立成功大學電機工程學系 - 電腦與通信工程研究所
> 高效能平行/分散系統實驗室 (HPDS Lab)
>
> National Cheng Kung University, Engineering Science Dpt.
> 國立成功大學工程科學系
>
> Contacts
> tzuli...@ee.ncku.edu.tw
> http://tzulitai.ee.ncku.edu.tw
> Linkedin: tw.linkedin.com/in/tzulitai
> +886981916890
>
Regards,
Chiwan Park
che Flink grow for
> this year:
>
> https://flink.apache.org/news/2015/12/18/a-year-in-review.html
>
> Happy New Year everyone and thanks for being part of this great community!
>
>
> Thanks,
>
> - Henry
Regards,
Chiwan Park
Great! Thanks for addressing!
> On Jan 6, 2016, at 5:51 PM, Stephan Ewen <se...@apache.org> wrote:
>
> At a first look, I think that "flink-runtime" does not need Apache Httpclient
> at all. I'll try to simply remove that dependency...
>
> On Wed, Jan 6, 201
be updated to announce latest stable
version to newcomers.
Is there any problem to update doc?
Regards,
Chiwan Park
2016 at 10:25 AM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> I think we missed updating the variable "version" in the "docs/_config.yml"
> for the 0.10.1 release.
>
> Would be good to update it and push a new version of the docs.
>
> Greeting
ext](
> new TextInputFormat,
> classOf[LongWritable],
> classOf[Text],
> new JobConf()
> ))
>
> The Java version is very similar.
>
> Note: Flink has wrappers for both MR APIs: mapred and mapreduce.
>
> Cheers,
> Fabian
>
> 2015-11-24 19:3
il.com> wrote:
>
> I completely missed this, thanks Chiwan. Can these be used with DataStreams
> as well as DataSets?
>
> On Tue, Nov 24, 2015 at 10:06 AM, Chiwan Park <chiwanp...@apache.org> wrote:
> Hi Nick,
>
> You can use Hadoop Input/Output Fo
gt; wrote:
>
> Hello,
>
> Is it possible to use existing Hadoop Input and OutputFormats with Flink?
> There's a lot of existing code that conforms to these interfaces, seems a
> shame to have to re-implement it all. Perhaps some adapter shim..?
>
> Thanks,
> Nick
Regards,
Chiwan Park
Hi all,
I think we can use `readFile`, `readFileStream` methods in
`StreamExecutionEnvironment` to create streaming source from S3 because data
are stored as file in S3. But I haven’t test it.
Regards,
Chiwan Park
> On Jun 3, 2016, at 2:37 PM, Tzu-Li (Gordon) Tai <tzuli...@gmail.com&
Hi Lydia,
`FlinkMLTools.persist` method is used to save ML models and can be used to save
Matrix and Vector object. Note that the method uses TypeSerializerOutputFormat
which is binary output format.
Regards,
Chiwan Park
> On May 30, 2016, at 11:31 AM, Lydia Ickler <ickle...@googlema
I’ve merged a patch [1] for this issue. Now we can use Option as a key.
[1]:
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=c60326f85faaa38bcc359d555cd2d2818ef2e4e7
Regards,
Chiwan Park
> On Apr 5, 2016, at 2:08 PM, Chiwan Park <chiwanp...@apache.org> wrote:
>
&g
. `RocksDBStateBackend` uses memory first and also
can spill states to disk.
Regards,
Chiwan Park
> On Jun 22, 2016, at 3:27 PM, Tae-Geon Um <taegeo...@gmail.com> wrote:
>
> I have another question.
> Is the spilling only executed on batch mode?
> What happen on streaming mode?
community has a plan [1] to move data structures for streaming
operators to managed memory.
[1]:
https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-778DAD7eqw5GANwE/edit#
Regards,
Chiwan Park
> On Jun 22, 2016, at 8:39 PM, Tae-Geon Um <taegeo...@gmail.com> wrote:
>
ESS [ 10.224
> s]
> [INFO] flink-runtime .. FAILURE [01:23
> min]
> [INFO] flink-optimizer SKIPPED
>
>
> Any workaround for scala_2.11.4 or do I have to switch back to scala_2.10.4 ?
>
> Thanks,
> Ritesh Kumar Singh,
> https://riteshtoday.wordpress.com/
>
Regards,
Chiwan Park
I wrote a sample inherited POJO example [1]. The example works with Flink
0.10.1 and 1.0-SNAPSHOT.
[1]: https://gist.github.com/chiwanpark/0389ce946e4fff58d611
Regards,
Chiwan Park
> On Feb 9, 2016, at 8:07 PM, Fabian Hueske <fhue...@gmail.com> wrote:
>
> What is the ty
Hi Dominique,
It seems that `SourceA` is not dealt as POJO. Are all fields in SourceA public?
There are some requirements for POJO classes [1].
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#pojos
Regards,
Chiwan Park
> On Feb 9, 2016, at 7
Oh, the fields in SourceA have public getters. Does the fields in SourceA have
public setter? SourceA needs public setter for private fields.
Regards,
Chiwan Park
> On Feb 9, 2016, at 7:45 PM, Chiwan Park <chiwanp...@apache.org> wrote:
>
> Hi Dominique,
>
> I
+names+suffixed+with+Scala+version
Regards,
Chiwan Park
> On Feb 10, 2016, at 9:39 AM, shotte <hotte.sylv...@gmail.com> wrote:
>
> Do I need to go to Flink 1.0 or the downgrade to Kafka 0.8 ?
>
>
>
> --
> View this message in context:
> http://apache-flink-use
Hi David,
I just downloaded the "flink-1.0-SNAPSHOT-bin-hadoop2_2.11.tgz” but there is no
jar compiled with Scala 2.10. Could you check again?
Regards,
Chiwan Park
> On Feb 10, 2016, at 2:59 AM, David Kim <david@braintreepayments.com>
> wrote:
>
> Hello,
>
The documentation I sent is for Flink 1.0.
In Flink 0.10.x, there is no suffix of dependencies for Scala 2.10 (e.g.
flink-streaming-java). But there is a suffix of dependencies for Scala 2.11
(e.g. flink-streaming-java_2.11).
Regards,
Chiwan Park
> On Feb 10, 2016, at 1:46 PM, Chiwan P
There is a JIRA issue (FLINK-1873, [1]) that covers the distributed matrix
implementation.
[1]: https://issues.apache.org/jira/browse/FLINK-1873
Regards,
Chiwan Park
> On Jan 27, 2016, at 5:21 PM, Chiwan Park <chiwanp...@apache.org> wrote:
>
> I hope the distributed ma
#transformations
Regards,
Chiwan Park
> On Jan 30, 2016, at 6:43 PM, LINZ, Arnaud <al...@bouyguestelecom.fr> wrote:
>
> Hello,
>
> I have a very big dataset A to left join with a dataset B that is half its
> size. That is to say, half of A records will be matched with one recor
,
Chiwan Park
[1]:
https://hive.apache.org/javadocs/r0.13.1/api/ql/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.html
[2]:
http://stackoverflow.com/questions/22673222/how-do-you-use-orcfile-input-output-format-in-mapreduce
[3]:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/api
Thanks for sharing, Ritesh!
Regards,
Chiwan Park
> On Jan 21, 2016, at 12:28 AM, Ritesh Kumar Singh
> <riteshoneinamill...@gmail.com> wrote:
>
> Thanks for the update Robert, I tried it out and it works fine for
> scala_2.11.4 version.
> I've made a docker image o
in SelectNearestCenter class, euclideanDistance method is used to
measure the distance between each point. For your implementation, you have to
substitute type to your data type (It can be your custom class or
Flink-provided Tuple) and change distance metric for your data.
Regards,
Chiwan Park
> On Jan
to the Flink blog [3].
Regards,
Chiwan Park
[1]: https://cwiki.apache.org/confluence/display/FLINK/1.0+Release
[2]:
https://cwiki.apache.org/confluence/display/FLINK/Maven+artifact+names+suffixed+with+Scala+version
[3]: http://flink.apache.org/blog/
> On Feb 17, 2016, at 3:34 PM, wangzhijiang
We’re testing a release candidate for 1.0 [1] currently. You can use new
features I’m not sure because I’m not in PMC of Flink but I think we can
release in a month.
Regards,
Chiwan Park
[1]:
http://mail-archives.apache.org/mod_mbox/flink-user/201602.mbox/%3CCAGr9p8AkiT0CT_YBwMhHCUYmoC9Stw
Hi Timur,
Great! Bootstrap action for Flink is good for AWS users. I think the bootstrap
action scripts would be placed in `flink-contrib` directory.
If you want, one of people in PMC of Flink will be assign FLINK-1337 to you.
Regards,
Chiwan Park
> On Apr 6, 2016, at 3:36 AM, Timur Fayru
Hi subash,
You can pass WriteMode in second parameter of write* method. For example:
```
DataStream<…> myStream = …;
myStream.writeAsCsv(“path of output”, FileSystem.WriteMode.OVERWRITE);
```
I hope this helps.
Regards,
Chiwan Park
> On Mar 22, 2016, at 8:18 PM, subash basn
.
Regards,
Chiwan Park
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/iterations.html
> On Mar 27, 2016, at 7:16 AM, Lydia Ickler <ickle...@googlemail.com> wrote:
>
> Hi,
>
> I have an issue with a for-loop.
> If I set the maximal iteration number
object.
Regards,
Chiwan Park
[1]:
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/sink/RichSinkFunction.html
> On Mar 7, 2016, at 10:08 PM, tole...@toletum.org wrote:
>
> Hi!
> I'm doing a process which reads from kafka, makes
]:
https://ci.apache.org/projects/flink/flink-docs-master/setup/building.html#scala-versions
[2]:
https://cwiki.apache.org/confluence/display/FLINK/Maven+artifact+names+suffixed+with+Scala+version
Regards,
Chiwan Park
> On Apr 5, 2016, at 9:40 AM, Andrew Gaydenko <andrew.gayde...@gmail.com&
I just found that Timur created a JIRA issue for this (FLINK-3698).
Regards,
Chiwan Park
> On Mar 31, 2016, at 7:27 PM, Till Rohrmann <trohrm...@apache.org> wrote:
>
> Actually I think that it’s not correct that the OptionType cannot be used as
> a key type. In
MyKey, right: MyKey) => (left, right)
}.print()
```
Note that the approach in example (using hashCode()) cannot be applied to sort
task.
Regards,
Chiwan Park
> On Mar 30, 2016, at 2:37 AM, Timur Fayruzov <timur.fairu...@gmail.com> wrote:
>
> There is some more detail to this question
original value). So there is some performance decrease when you
are using KeySelector.
Regards,
Chiwan Park
> On Mar 31, 2016, at 12:58 AM, Timur Fayruzov <timur.fairu...@gmail.com> wrote:
>
> Thank you Chiwan! Yes, I understand that there are workarounds that don't use
> fun
1) {
(left, right) => 1
}
```
I hope this helps.
[1]:
https://ci.apache.org/projects/flink/flink-docs-master/apis/common/index.html#define-keys-for-tuples
Regards,
Chiwan Park
> On Mar 30, 2016, at 3:54 AM, Timur Fayruzov <timur.fairu...@gmail.com> wrote:
>
> Hello,
>
`.
Regards,
Chiwan Park
> On Apr 28, 2016, at 9:51 AM, nsengupta <sengupta.nirma...@gmail.com> wrote:
>
> Hello Chiwan,
>
> Yes, that's an oversight on my part. In my hurry, I didn't even try to
> explore the source of that /Exception/. Thanks, again.
>
> However, I still
headers by calling zipWithIndex method and filtering it based on the index.
Regards,
Chiwan Park
> On Apr 27, 2016, at 10:32 AM, nsengupta <sengupta.nirma...@gmail.com> wrote:
>
> What is the recommended way of discarding the Column Header(s) from a CSV
> f
Hi,
You don’t need to call execute() method after calling print() method. print()
method triggers the execution. The exception is raised because you call
execute() after print() method.
Regards,
Chiwan Park
> On Apr 27, 2016, at 6:35 PM, nsengupta <sengupta.nirma...@gmail.com&
Hi Jon,
You can connect Flink Web UI via clicking ApplicationMaster link in YARN
administrator UI.
Regards,
Chiwan Park
> On Aug 15, 2016, at 2:24 PM, Jon Yeargers <jon.yearg...@cedexis.com> wrote:
>
> Working with a 3 node cluster. Started via YARN.
>
> If I
77 matches
Mail list logo
