Github user hntd187 commented on the pull request:
https://github.com/apache/spark/pull/1290#issuecomment-111337485
@avulanov Can spark even read an HDF5 file or would we have to write that
as well? While, I can't donate any professional time to this conversion
problem, but I may be able to assist if we wanted to write a conversion
independently. I suppose the problem here, is even if we get HDF5 and run it in
caffe, how would we get spark to use it? Reading a bit online and looking
around, it seems to be the consensus to use the pyhdf5 library to read the
files in and do a flatMap to RDDs, but that would seem horribly inefficient on
a large dataset and we'd be shooting yourselves in the foot trying to make that
scale. So I think, our best bet is if we want to compare to caffe is either,
get caffe to read another format or add HDF5 reading capability to spark either
via a hack or an actual contribution. First one is not ideal, second one is
obviously more time consuming.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
