sorry meant to say:
we know when we upgrade that we might run into minor inconveniences that
are completely our own doing/fault.
also, with yarn it has become really easy to run against an exact spark
version of our choosing, since there is no longer such a thing as a
centrally managed spark distr
i agree with that.
we work within that assumption. we compile and run against a single exact
spark version. we know when we upgrade that we might run into minor
inconveniences that our completely our own doing/fault. the trade off has
been totally worth it to me.
On Thu, Mar 30, 2017 at 1:20 PM,
I think really the right way to think about things that are marked private
is, "this may disappear or change in a future minor release". If you are
okay with that, working about the visibility restrictions is reasonable.
On Thu, Mar 30, 2017 at 5:52 AM, Koert Kuipers wrote:
> I stopped asking l
I stopped asking long time ago why things are private in spark... I mean...
The conversion between ml and mllib vectors is private... the conversion
between spark vector and breeze used to be (or still is?) private. it just
goes on. Lots of useful stuff is private[SQL].
Luckily there are simple wa
spark version 2.1.0, vector is from ml package. the Vector in mllib has a
public VectorUDT type
On Thu, Mar 30, 2017 at 10:57 AM, Ryan wrote:
> I'm writing a transformer and the input column is vector type(which is the
> output column from other transformer). But as the VectorUDT is private, how
I'm writing a transformer and the input column is vector type(which is the
output column from other transformer). But as the VectorUDT is private, how
could I check/transform schema for the vector column?