Thanks for pointing to the yarn JIRA. For now, it would be good for my talk since it brings out that hadoop and big data community is already aware of the GPUs and making effort to exploit it.
Good luck for your talk. That fear is lurking in my mind too :) On 10-Sep-2015 2:08 pm, "Steve Loughran" <ste...@hortonworks.com> wrote: > > > On 9 Sep 2015, at 20:18, lonikar <loni...@gmail.com> wrote: > > > > I have seen a perf improvement of 5-10 times on expression evaluation > even > > on "ordinary" laptop GPUs. Thus, it will be a good demo along with some > > concrete proposals for vectorization. As you said, I will have to hook > up to > > a column structure and perform computation and let the existing spark > > computation also proceed and compare the performance. > > > > you might also be interested to know that there's now a YARN JIRA on > making GPU another resource you can ask for > https://issues.apache.org/jira/browse/YARN-4122 > > if implemented, it'd let you submit work into the cluster asking for GPUs, > and get allocated containers on servers with the GPU capacity you need. > This'd allow you to share GPUs with other code (including your own > containers) > > > I will focus on the slides early (7th Oct is deadline), and then continue > > the work for another 3 weeks till the summit. It still gives me enough > time > > to do considerable work. Hope your fear does not come true. > > good luck. And the fear is about my talk at apachecon on the Hadoop stack > & Kerberos > > > >
