hi ben you can mimic a yarn instance by creating a slave and a master. this would confirm no serialization problem are involved
Le 28 sept. 2017 à 16:55, Benedict Holland écrivait : > Hello All, > > It does, in fact, look like it works with standalone instances. We don't > have an environment to test with yarn, but given how it works, it looks > like it should work fine. The only thing is, each node will have to have > access to the database that the CPE runs over. I was actually thinking > about making the Dataset<Row> collection be created from the CPE getNext() > method until hasNext returns false, but I think that will cause memory > problems with huge databases. > > Hopefully, I will have more information on exactly what I can release over > the next upcoming days. I am pushing to provide a minimum working example > with a MySQL schema and a small setup guide. > > ~Ben > > On Thu, Sep 28, 2017 at 3:29 AM, Nicolas Paris <[email protected]> wrote: > > > Hey ben > > > > thanks for the feedbacak, looks interesting approach > > have you both validate your approach on standalone/yarn spark instances > > ? > > > > thanks > > Le 26 sept. 2017 à 21:02, Benedict Holland écrivait : > > > Hello all, > > > > > > I have a working application that essentially implements the CPE within a > > > spark context. The best part about this is that it does not use UIMAFit > > or > > > any 3rd party applications. It simply uses hadoop, spark, UIMA, and > > > OpenNLP. > > > > > > Users are able to configure, design, and build the UIMA pipeline using > > all > > > of the eclipse XML plugin applications. Instead of running the > > application > > > via the CPE.process() driver from a main class, it will run from the > > > foreach() function on the Dataframe<Row> object. > > > > > > Oh also, it plugs into a database to get the text and to write results. > > > > > > Would the UIMA community be interested in getting a working example put > > > together? If so, please feel free to contact me. I think this could be an > > > excellent example of what people would like to use and your examples are > > > particularly good. > > > > > > Thanks, > > > ~Ben > >
