That's a good idea. We didn't intentionally break the doc generation. The
doc generation for Catalyst is broken because we use Scala macros and we
haven't had time to investigate how to fix it yet.
If you have a minute and want to investigate, I can merge it in as soon as
possible.
On Fri, Ja
Reynold,
Your clarification is much appreciated. One issue though, that I would
strongly encourage you to work on, is to make sure that the Scaladoc CAN be
generated manually if needed (a "Use at your own risk" clause would be
perfectly legitimate here). The reason I say this is that currently eve
We can look into some sort of util class in sql.types for general type
inference. In general many methods in JsonRDD might be useful enough to
extract. Those will probably be marked as DeveloperAPI with less stability
guarantees.
On Thu, Jan 15, 2015 at 12:16 PM, Corey Nolet wrote:
> Reynold,
>
Reynold,
One thing I'd like worked into the public portion of the API is the json
inferencing logic that creates a Set[(String, StructType)] out of
Map[String,Any]. SPARK-5260 addresses this so that I can use Accumulators
to infer my schema instead of forcing a map/reduce phase to occur on an RDD
Alex,
I didn't communicate properly. By "private", I simply meant the expectation
that it is not a public API. The plan is to still omit it from the
scaladoc/javadoc generation, but no language visibility modifier will be
applied on them.
After 1.3, you will likely no longer need to use things in
Reynold,
Thanks for the heads up. In general, I strongly oppose the use of "private"
to restrict access to certain parts of the API, the reason being that I
might find the need to use some of the internals of a library from my own
project. I find that a @DeveloperAPI annotation serves the same pur
Hi Spark devs,
Given the growing number of developers that are building on Spark SQL, we
would like to stabilize the API in 1.3 so users and developers can be
confident to build on it. This also gives us a chance to improve the API.
In particular, we are proposing the following major changes. Thi
