[
https://issues.apache.org/jira/browse/MAHOUT-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233804#comment-15233804
]
ASF GitHub Bot commented on MAHOUT-1750:
----------------------------------------
GitHub user andrewpalumbo opened a pull request:
https://github.com/apache/mahout/pull/215
MAHOUT-1750 - For Review. FlinkOpABt too many operations to implement?
When (partially) Implementing `FlinkOpABt` so far, It seems that there may
be too many Flink map/reduce/group/etc operations to finish using this method.
**NOTE**: this is unfinished and is not numerically correct.
Currently, when testing am getting kyro stack overflow exceptions, which as
I understand are often caused by a long string of operations:
```
Job execution failed.
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:716)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.StackOverflowError
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:74)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:57)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
{...}```
Any comments are appreciated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1750
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/mahout/pull/215.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #215
----
commit fab6bb86fa1af0a7efce2300dcada45c32b35677
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-28T01:34:34Z
start Stealing DL's Spark code
commit 7e1a8bbaaca71036e9c0247f07a40368432f592b
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-28T01:55:01Z
wip: Spark combineByKey => Flink groupReduce
commit 31a644b36ed92c5aae3b80d5e8fbca7dd8a1d87a
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-28T22:38:31Z
implement first step of the combiner
commit 7dc73890ac5eeaad1ba38ab799b19b989c6ff918
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-28T22:53:09Z
Add in type information for maps, etc.
commit 5c60167f92a32b4c0751f6bf8dc12a838e1cde78
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-28T23:50:12Z
Merge branch 'flink-binding' of
https://git-wip-us.apache.org/repos/asf/mahout into MAHOUT-1750
commit 9b01a4a498c080e6063387608253890a13e77ad4
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-29T02:48:43Z
remove all unnecessary type info still not builing with
GroupCombineFunctions
commit 37391a5a51b6803fdf893349a45a61903a6928e5
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-30T00:07:26Z
wip
commit d65e23b91f3c055c80de432dc9699c62512dd34e
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-30T00:07:40Z
Merge branch 'flink-binding' of
https://git-wip-us.apache.org/repos/asf/mahout into MAHOUT-1750
commit 3127d3218d9fa0fba35c9c240db63e71b00c8a32
Author: Andrew Palumbo <[email protected]>
Date: 2016-03-30T01:01:15Z
wip: still problems with error: object creation impossible, since method
mapPartition in class RichMapPartitionFunction of type (x: Iterable[(Array[K1],
org.apache.mahout.math.Matrix)], x: org.apache.flink.util.Collector[(Int,
Array[K1], org.apache.mahout.math.Matrix)])Unit is not defined
commit c19493b058e6d92f10fadeb32be9bafb14e7c671
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T05:17:03Z
WIP
commit b9ed381c2178392e785c4835adfac53b8d0ec5fb
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T08:41:58Z
Revert "WIP"
This reverts commit c19493b058e6d92f10fadeb32be9bafb14e7c671.
commit 3311e89fbef3751476dbf113f680330b61f64160
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T08:51:20Z
Merge branch 'flink-binding' of
https://git-wip-us.apache.org/repos/asf/mahout into MAHOUT-1750
commit db21e7b12a0b4e05ff27fa9031fd8bf3e5f22bda
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T17:02:05Z
Use java.lang.Iterable instead of scala.Iterable- this fixes the object
creation issue.
commit d3d60fadf50ceb12f559d4c94c9402735c8a7d30
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T21:45:03Z
wip
commit 0a83e122c82eb0584a00510f7fb571432e834064
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T22:11:26Z
wip: error is
Error:(180, 23) overriding method reduce in trait ReduceFunction of type
(x: (Array[K], org.apache.mahout.math.Matrix), x: (Array[K],
org.apache.mahout.math.Matrix))(Array[K], org.apache.mahout.math.Matrix);
method reduce has incompatible type
def reduce(mx1: (Array[K], Matrix), mx2: (Array[K],
Matrix)) {
^
commit 97a6aea7c4fd0abc719605991322ac4acb758ea8
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-08T22:20:56Z
wip: compiling now
commit 088d925ba3cc5e93e25ad37f8e2fcf4d4698595f
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-09T00:44:22Z
not returning any partitions. failing at dimensionality count in
CheckpointedDrmFlink
commit 82ae321b85b84e478152f208059f9c8146d814c8
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-09T01:01:05Z
wip: need to make combiner into a merger
commit 3c7c2ff937206b0882f6395058a8646ed1c0cdaa
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-09T14:29:38Z
wip: comments, combiner
commit cd2b59ac96b63c93ae18619cdf22b6c2742e93d3
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-09T19:58:29Z
wip: kryo error
commit d313ebc1618ae5cb44b5161b492bb03baba8216d
Author: Andrew Palumbo <[email protected]>
Date: 2016-04-10T00:58:13Z
Kryo stackOverflow
----
> Mahout DSL for Flink: Implement ABt
> -----------------------------------
>
> Key: MAHOUT-1750
> URL: https://issues.apache.org/jira/browse/MAHOUT-1750
> Project: Mahout
> Issue Type: Task
> Components: Math
> Affects Versions: 0.10.2
> Reporter: Alexey Grigorev
> Assignee: Andrew Palumbo
> Priority: Minor
> Fix For: 0.12.0
>
>
> Now ABt is expressed through AtB, which is not optimal, and we need to have a
> special implementation for ABt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
