I think that is enough -1's that we should sink this release and get this test figured out. I will spend some time looking at it this morning. Can some others also lend a hand?
If we can't figure out why the impersonation test is causing problems (or a solution), I'm inclined to disable it and stop holding up the release. It definitely seems like it probably has more to do with the test structure itself rather than the a Drill bug. I'll shoot to start a new release candidate vote later today. In terms of master, I propose that we fork a 1.3.0 release branch at commit e78e28661e921c6bd07ba7bb95d91f9adeb5a247 and then we can reopen changes on master. thanks, Jacques -- Jacques Nadeau CTO and Co-Founder, Dremio On Mon, Nov 9, 2015 at 6:14 PM, Jacques Nadeau <[email protected]> wrote: > Anybody want to profile what the impersonation tests are doing that are > causing so many problems with resources? > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Mon, Nov 9, 2015 at 5:24 PM, Hanifi Gunes <[email protected]> wrote: > >> 1024/4096/2032195 <==> soft/hard/system >> >> On Mon, Nov 9, 2015 at 4:50 PM, Jacques Nadeau <[email protected]> >> wrote: >> >> > What about open files? >> > On Nov 9, 2015 4:20 PM, "Hanifi Gunes" <[email protected]> wrote: >> > >> > > Using the default parallelism, I get the same set of exceptions on >> > CentOS. >> > > >> > > Also max # of threads per process is the system default, 319548. >> > > >> > > On Mon, Nov 9, 2015 at 4:13 PM, Amit Hadke <[email protected]> >> wrote: >> > > >> > > > Unfortunately, yes I changed them at OS level. >> > > > >> > > > I agree that making OS level changes to run tests is not something >> we >> > > > should advocate. >> > > > It was just an observation that tests seem to pass on linux with >> > default >> > > > settings but on mac behavior is irregular. >> > > > >> > > > ~ Amit. >> > > > >> > > > >> > > > On Mon, Nov 9, 2015 at 3:48 PM, Jason Altekruse < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > Amit, >> > > > > >> > > > > Did you change these at the OS level? I would hope that we >> wouldn't >> > > have >> > > > to >> > > > > make people jump through such hoops to run the build, was there a >> > > thread >> > > > I >> > > > > missed where we discussed this something that we should be >> expecting >> > > > > everyone to do? I did see the threads about increasing the maven >> > > permgen >> > > > > and did so accordingly, but I think adjusting something like the >> > system >> > > > > ulimit is something we really should try to avoid. >> > > > > >> > > > > On Mon, Nov 9, 2015 at 3:44 PM, Abdel Hakim Deneche < >> > > > [email protected] >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > I just tried with the default forkCount and got similar >> failures. >> > I'm >> > > > > > running the tests on a linux machine with the default maven >> options >> > > > > > >> > > > > > Thanks >> > > > > > >> > > > > > On Mon, Nov 9, 2015 at 3:21 PM, Jacques Nadeau < >> [email protected] >> > > >> > > > > wrote: >> > > > > > >> > > > > > > So you're changing the forkCount from the default? What >> happens >> > > when >> > > > > you >> > > > > > > run with the default? >> > > > > > > >> > > > > > > Also, did you guys run with extra permgen or the default? >> Also, >> > can >> > > > you >> > > > > > > confirm what type of machine you are running on? >> > > > > > > >> > > > > > > Want to figure out why things are different. >> > > > > > > >> > > > > > > thanks, >> > > > > > > Jacques >> > > > > > > >> > > > > > > -- >> > > > > > > Jacques Nadeau >> > > > > > > CTO and Co-Founder, Dremio >> > > > > > > >> > > > > > > On Mon, Nov 9, 2015 at 2:23 PM, Abdel Hakim Deneche < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Saw similar failures when running unit tests with forkCount >> = >> > 1: >> > > > > > > > >> > > > > > > > Tests in error: >> > > > > > > > > >> > > > > >> TestImpersonationQueries.sequenceFileChainedImpersonationWithView » >> > > > > > > > > UserRemote >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationJoinEachSideReachesMaxUserHops:233->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » Rpc >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationExceedsMaxUserHops:219->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.avroChainedImpersonationWithView:280->BaseTestImpersonation.createView:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testDirectImpersonation_HasGroupReadPermissions:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testDirectImpersonation_NoReadPermissions:196->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationEqualToMaxUserHops:210->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > > > > » IllegalState >> > > > > > > > > TestCountDownLatchInjection.latchInjected:146 >> NullPointer >> > > > > > > > > TestMergingReceiver.testMultipleProvidersMixedSizes:115 >> » >> > > > > > > > > IllegalReferenceCount >> > > > > > > > > Tests run: 1482, Failures: 0, Errors: 9, Skipped: 118 >> > > > > > > > >> > > > > > > > >> > > > > > > > Unfortunately I'm changing my vote to -1 >> > > > > > > > >> > > > > > > > >> > > > > > > > On Mon, Nov 9, 2015 at 2:21 PM, Hanifi Gunes < >> > > [email protected]> >> > > > > > > wrote: >> > > > > > > > >> > > > > > > > > -1 (binding) >> > > > > > > > > >> > > > > > > > > 5/5 consecutive runs I get the following consistently with >> > > > > forkCount >> > > > > > > set >> > > > > > > > to >> > > > > > > > > 1 on build. >> > > > > > > > > >> > > > > > > > > Tests in error: >> > > > > > > > > >> > > > > >> TestImpersonationQueries.sequenceFileChainedImpersonationWithView » >> > > > > > > > > UserRemote >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationJoinEachSideReachesMaxUserHops:233->BaseTestQuery.updateClient:222->BaseTestQuery. >> > > > > > > > > updateClient:236->BaseTestQuery.updateClient:213 » >> > > > IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationExceedsMaxUserHops:219->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient: >> > > > > > > > > 236->BaseTestQuery.updateClient:213 » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.avroChainedImpersonationWithView:280->BaseTestImpersonation.createView:186->BaseTestQuery.updateClient:222- >> > > > > > > > > >> > > > > >BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213 >> > > > > > » >> > > > > > > > > IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testDirectImpersonation_HasGroupReadPermissions:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient: >> > > > > > > > > 236->BaseTestQuery.updateClient:213 » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testDirectImpersonation_NoReadPermissions:196->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236- >> > > > > > > > > >BaseTestQuery.updateClient:213 » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestImpersonationQueries.testMultiLevelImpersonationEqualToMaxUserHops:210->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient: >> > > > > > > > > 236->BaseTestQuery.updateClient:213 » IllegalState >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Mon, Nov 9, 2015 at 2:19 PM, Stefán Baxter < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi, >> > > > > > > > > > >> > > > > > > > > > I'm not a committer and not voting but I want to point >> out >> > > that >> > > > > the >> > > > > > > > UDFs >> > > > > > > > > > that we are using and run with 1.1 and 1.2 do not, for >> some >> > > > > reason, >> > > > > > > run >> > > > > > > > > > with 1.3. >> > > > > > > > > > >> > > > > > > > > > I'm inclined to insist that it's something we are doing >> > wrong >> > > > but >> > > > > > if >> > > > > > > > this >> > > > > > > > > > counts as a test then it's failing. >> > > > > > > > > > >> > > > > > > > > > Regards, >> > > > > > > > > > -Stefan >> > > > > > > > > > >> > > > > > > > > > On Mon, Nov 9, 2015 at 10:09 PM, Norris Lee < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > +1 Non-binding. >> > > > > > > > > > > >> > > > > > > > > > > Built from source on Linux, tested ODBC against >> various >> > > data >> > > > > > > sources >> > > > > > > > > > > (Hive, DFS-csv, tsv, parquet, json) >> > > > > > > > > > > >> > > > > > > > > > > Norris >> > > > > > > > > > > >> > > > > > > > > > > -----Original Message----- >> > > > > > > > > > > From: Abdel Hakim Deneche [mailto: >> [email protected]] >> > > > > > > > > > > Sent: Monday, November 09, 2015 10:16 AM >> > > > > > > > > > > To: [email protected] >> > > > > > > > > > > Subject: Re: [VOTE] Release Apache Drill 1.3.0 (rc1) >> > > > > > > > > > > >> > > > > > > > > > > +1 >> > > > > > > > > > > >> > > > > > > > > > > Checked checksum files and gpg signature for all 3 >> files. >> > > > Built >> > > > > > > from >> > > > > > > > > > > source, deployed on a 4 nodes cluster then run several >> > > window >> > > > > > > > function >> > > > > > > > > > > queries on TPCDS SF100. Looks great. >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Nov 9, 2015 at 8:14 AM, Jacques Nadeau < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > I've seen another instance of DRILL-4041. Since it >> is >> > > > > > exceedingly >> > > > > > > > > > > > rare, I don't think it is a release stopper. I will >> > spend >> > > > > some >> > > > > > > more >> > > > > > > > > > > > time reviewing it. >> > > > > > > > > > > > >> > > > > > > > > > > > -- >> > > > > > > > > > > > Jacques Nadeau >> > > > > > > > > > > > CTO and Co-Founder, Dremio >> > > > > > > > > > > > >> > > > > > > > > > > > On Sun, Nov 8, 2015 at 7:34 PM, Amit Hadke < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > +1, non-binding >> > > > > > > > > > > > > Downloaded tarball, mvn clean install Ran join >> > > queries,, >> > > > > was >> > > > > > > able >> > > > > > > > > to >> > > > > > > > > > > > > run queries on json, parquet and sequence files. >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Sun, Nov 8, 2015 at 10:01 AM, Abhijit Pol < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > resending the vote with details >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > +1, non-binding >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > ran maven tests, installed on Mac, ran drill in >> > > > embedded >> > > > > > > mode, >> > > > > > > > > > > > > > verified >> > > > > > > > > > > > > Web >> > > > > > > > > > > > > > UI for CSV header config changes, ran few >> queries >> > on >> > > > CSV >> > > > > > > files >> > > > > > > > > > > > > > with and without header, ran few queries against >> > > large >> > > > > CSV >> > > > > > > > files >> > > > > > > > > > > > > > with header in >> > > > > > > > > > > > > HDFS >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sun, Nov 8, 2015 at 9:07 AM, Hsuan Yi Chu < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > +1 >> > > > > > > > > > > > > > > mvn clean install on mac and linux >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > tried a few queries on distributed mode. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Things looked fine >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Sun, Nov 8, 2015 at 8:16 AM, Aman Sinha < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > +1. >> > > > > > > > > > > > > > > > Downloaded the binary tar ball and >> installed on >> > > > Mac. >> > > > > > Ran >> > > > > > > > > > > > > > > > Drill in >> > > > > > > > > > > > > > > embedded >> > > > > > > > > > > > > > > > mode. Tested a few join, aggregation and >> limit >> > > > > queries >> > > > > > > > > > > > > > > > against >> > > > > > > > > > > > > parquet >> > > > > > > > > > > > > > > > data with and without metadata cache. >> Tested >> > > > > > > cancellation >> > > > > > > > > for >> > > > > > > > > > > > couple >> > > > > > > > > > > > > > of >> > > > > > > > > > > > > > > > queries. Looked at query profiles on the >> > WebUI. >> > > > > > Looks >> > > > > > > > > good. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Sat, Nov 7, 2015 at 11:53 PM, Abhijit Pol >> > > > > > > > > > > > > > > > <[email protected]> >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > +1 >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Sat, Nov 7, 2015 at 9:41 PM, andrew < >> > > > > > > > [email protected]> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > +1 (non-binding, non-comitter) >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Nov 6, 2015, at 10:15 PM, Jacques >> > > Nadeau < >> > > > > > > > > > > > > [email protected]> >> > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Hey Everybody, >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > I'd like to propose a new release >> > candidate >> > > > of >> > > > > > > Apache >> > > > > > > > > > > > > > > > > > > Drill, >> > > > > > > > > > > > > > > version >> > > > > > > > > > > > > > > > > > > 1.3.0. This is the second release >> > > candidate >> > > > > > (rc1). >> > > > > > > > > > > > > > > > > > > This >> > > > > > > > > > > > > > addresses >> > > > > > > > > > > > > > > > > some >> > > > > > > > > > > > > > > > > > > issues identified in the first release >> > > > > candidate >> > > > > > > > > > > > > > > > > > > including >> > > > > > > > > > > > some >> > > > > > > > > > > > > > > test >> > > > > > > > > > > > > > > > > > > threading issues, Parquet dictionary >> read >> > > > issue >> > > > > > > with >> > > > > > > > > > > > > > > > > > > binary >> > > > > > > > > > > > > > values, >> > > > > > > > > > > > > > > > > Joda >> > > > > > > > > > > > > > > > > > > upgrade to avoid a concurrency bug >> and a >> > > > couple >> > > > > > > other >> > > > > > > > > > > > > > > > > > > small >> > > > > > > > > > > > > > things. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > The tarball artifacts are hosted at >> [2] >> > and >> > > > the >> > > > > > > maven >> > > > > > > > > > > > artifacts >> > > > > > > > > > > > > > are >> > > > > > > > > > > > > > > > > > hosted >> > > > > > > > > > > > > > > > > > > at [3]. This release candidate is >> based >> > on >> > > > > commit >> > > > > > > > > > > > > > > > > > > >> f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305 >> > > > > located >> > > > > > at >> > > > > > > > > [4]. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > The vote will be open for 72 hours >> ending >> > > at >> > > > > 10PM >> > > > > > > > > > > > > > > > > > > Pacific, >> > > > > > > > > > > > > > November >> > > > > > > > > > > > > > > > 9, >> > > > > > > > > > > > > > > > > > 2015. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > [ ] +1 >> > > > > > > > > > > > > > > > > > > [ ] +0 >> > > > > > > > > > > > > > > > > > > [ ] -1 >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > thanks, >> > > > > > > > > > > > > > > > > > > Jacques >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > [1] >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > >> > > > > > >> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313 >> > > > > > > > > > > > 820&version=12332946[2] >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1 >> > > > > > > > > > > > > > > > > > > / >> > > > > > > > > > > > > > > > > > > [3] >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > >> > > > > > >> > > >> https://repository.apache.org/content/repositories/orgapachedrill-1014 >> > > > > > > > > > > > / >> > > > > > > > > > > > > > > > > > > [4] >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1 >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > > > > > Jacques Nadeau >> > > > > > > > > > > > > > > > > > > CTO and Co-Founder, Dremio >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > -- >> > > > > > > > > > > >> > > > > > > > > > > Abdelhakim Deneche >> > > > > > > > > > > >> > > > > > > > > > > Software Engineer >> > > > > > > > > > > >> > > > > > > > > > > <http://www.mapr.com/> >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > Now Available - Free Hadoop On-Demand Training < >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > >> > > > > > > > Abdelhakim Deneche >> > > > > > > > >> > > > > > > > Software Engineer >> > > > > > > > >> > > > > > > > <http://www.mapr.com/> >> > > > > > > > >> > > > > > > > >> > > > > > > > Now Available - Free Hadoop On-Demand Training >> > > > > > > > < >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > Abdelhakim Deneche >> > > > > > >> > > > > > Software Engineer >> > > > > > >> > > > > > <http://www.mapr.com/> >> > > > > > >> > > > > > >> > > > > > Now Available - Free Hadoop On-Demand Training >> > > > > > < >> > > > > > >> > > > > >> > > > >> > > >> > >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
