Oh, yes. I confused myself,... Should be /usr/lib...sorry! For that hive thing, I mean, if we can't have a proper working spark env w/o datanucleus, we can only accept it and put them together even though I don't use hive. This should be a spark issue we can't fix it at our side.
Merge spark-datanucleus into spark-core seems reasonable given the current context, but maybe in the future spark will have a better decoupled code? It's much more comfortable to me to apply a workaround and expect they can be decoupled in the future. 2016-01-24 9:35 GMT+08:00 Konstantin Boudnik <[email protected]>: > On Sun, Jan 24, 2016 at 09:22AM, Evans Ye wrote: > > I actually feels OK about having the dependency, if that's what spark > want. > > This isn't what Spark wants; this is what spark-hive integration needs. > But my > point is a bit different: let's suppose we all agree we need the Hive > support > in Spark (which I am not convinced, but don't care enough to make an > argument). Now, without datanucleus libs spark-shell is broken and won't > work. > Perhaps other clients using spark-submit won't work either (again, I don't > know nor care enough about Spark to figure out). Shall we simply get rid of > the datanucleus package and install these libs everywhere now? Cause you > know > - having a package just for the sake of having the package doesn't sound > very > optimal to me. See where I am going with it? > > > BTW, during the test I found that the spark installation dir are under > > /usr/lib/spark instead of /var/lib/spark. > > I expect we should put all the component libs under /var/lib for > > consistency. Is it correct? > > No, I believe /usr/lib/<component> is what we've been doing for a long > time. > Look at /usr/lib/hadoop or hbase or anything else for that matter. > > Cos > > > 2016-01-23 4:27 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > Hey all. > > > > > > I hate to pollute the list with this extra traffic, but there's a quite > > > ugly > > > issue that we are facing because of the infamous Hive-Spark collusion. > I am > > > talking about https://issues.apache.org/jira/browse/BIGTOP-2154 > > > > > > I have proposed a patch to work around it and let the 1.1 out of the > door. > > > I > > > have also open BIGTOP-2268 so we don't forget to fix it properly in the > > > next > > > release. > > > > > > I'd appreciate someone else taking a look and sharing the feedback, so > we > > > can > > > finally unblock the RC1. > > > > > > Thanks, > > > Cos > > > >
