Re: Extending Scala style checks

Since we can easily catch the list of all changed files in a PR, I think we can start with adding the no trailing space check for newly changed files only? On 10/2/14 9:24 AM, Nicholas Chammas wrote: Yeah, I remember that hell when I added PEP 8 to the build checks and fixed all the outstandin

Re: Extending Scala style checks

There is scalariform but it can be disruptive. Last time I ran it on Spark it didn't compile due to some xml interpolation problem. On Wednesday, October 1, 2014, Nicholas Chammas wrote: > Does anyone know if Scala has something equivalent to autopep8 > ? I

Re: Extending Scala style checks

Does anyone know if Scala has something equivalent to autopep8 ? It would help patch up the existing code base a lot quicker as we add in new style rules. ​ On Wed, Oct 1, 2014 at 9:24 PM, Nicholas Chammas wrote: > Yeah, I remember that hell when I added PE

Re: Extending Scala style checks

Yeah, I remember that hell when I added PEP 8 to the build checks and fixed all the outstanding Python style issues. I had to keep rebasing and resolving merge conflicts until the PR was merged. It's a rough process, but thankfully it's also a one-time process. I might be able to help with that in

Re: Extending Scala style checks

The hard part here is updating the existing code base... which is going to create merge conflicts with like all of the open PRs... On Wed, Oct 1, 2014 at 6:13 PM, Nicholas Chammas wrote: > Ah, since there appears to be a built-in rule for end-of-line whitespace, > Michael and Cheng, y'all should

Re: Extending Scala style checks

Ah, since there appears to be a built-in rule for end-of-line whitespace, Michael and Cheng, y'all should be able to add this in pretty easily. Nick On Wed, Oct 1, 2014 at 6:37 PM, Patrick Wendell wrote: > Hey Nick, > > We can always take built-in rules. Back when we added this Prashant > Sharm

Re: Extending Scala style checks

Hey Nick, We can always take built-in rules. Back when we added this Prashant Sharma actually did some great work that lets us write our own style rules in cases where rules don't exist. You can see some existing rules here: https://github.com/apache/spark/tree/master/project/spark-style/src/main

Re: Extending Scala style checks

Please take a look at WhitespaceEndOfLineChecker under: http://www.scalastyle.org/rules-0.1.0.html Cheers On Wed, Oct 1, 2014 at 2:01 PM, Nicholas Chammas wrote: > As discussed here , it would be > good to extend our Scala style checks to programmatica

Extending Scala style checks

As discussed here , it would be good to extend our Scala style checks to programmatically enforce as many of our style rules as possible. Does anyone know if it's relatively straightforward to enforce additional rules like the "no trailing spaces" rule me

Re: amplab jenkins is down

Sounds good! Thanks for the update Shane. On Wed, Oct 1, 2014 at 4:44 PM, shane knapp wrote: > as of this morning, i've got the new jenkins up, with all of the current > builds set up (but failing). i'm in the middle of playing setup/debug > whack-a-mole, but we're getting there. my guess woul

Re: amplab jenkins is down

as of this morning, i've got the new jenkins up, with all of the current builds set up (but failing). i'm in the middle of playing setup/debug whack-a-mole, but we're getting there. my guess would be early next week for the switchover. On Wed, Oct 1, 2014 at 12:53 PM, Nicholas Chammas < nicholas

Re: do MIMA checking before all test cases start?

How early can MiMa checks be run? Before Spark is even built ? After the build but before the unit tests? On Thu, Sep 25, 2014 at 6:06 PM, Patrick Wendell wrote: > Yeah we can also move it first. Wo

Re: amplab jenkins is down

On Thu, Sep 4, 2014 at 4:19 PM, shane knapp wrote: > on a side note, this incident will be accelerating our plan to move the > entire jenkins infrastructure in to a managed datacenter environment. > this > will be our major push over the next couple of weeks. more details about > this, also, as

Re: jenkins downtime/system upgrade wednesday morning, 730am PDT

upgrade complete! we're back online and happily building. On Wed, Oct 1, 2014 at 7:14 AM, shane knapp wrote: > jenkins is currently in quiet mode, and will be restarted once the current > crop of builds finishes. > > On Tue, Sep 30, 2014 at 2:40 PM, shane knapp wrote: > >> reminder: this is h

Re: jenkins downtime/system upgrade wednesday morning, 730am PDT

jenkins is currently in quiet mode, and will be restarted once the current crop of builds finishes. On Tue, Sep 30, 2014 at 2:40 PM, shane knapp wrote: > reminder: this is happening tomorrow morning. i will be putting jenkins > in to quiet mode at ~7am, and then doing the upgrade once any stra

Re: parquet predicate / projection pushdown into unionAll

Hi Cody and Michael, We ran into the same issue. Each day of data we have is stored into one parquet, and we want to query it against monthly parquet data. The data for each data is around 600GB, and we use 300 executors with 8GB memory for each executor. Without the patch, it took forever, and cr