Hi Tomas, We'd definitely welcome your contribution. I'm not sure if you found it but we have a wiki page that describes how to post a code review, etc.
I agree on the -march=native, I think the main obstacle is that to get good coverage of enough hardware configurations we'd need to distribute a lot of extra binaries with CDH. The other thing is that we try to do runtime code-generation for most perf-critical code, with some exceptions in the Parquet scanner, so the effect may not be as large as you'd typically expect. If I understand what you're asking, I think you'd need to rebuild Impala with some different compile options. We set them in the be/CMakeLists.txt file: https://github.com/cloudera/Impala/blob/cdh5-trunk/be/CMakeLists.txt#L42 On Sat, Jul 2, 2016 at 1:14 AM, Tomáš Gavenčiak <[email protected]> wrote: > I understand the SSSE3 warning then ... Perhaps a rewording of the message > (e.g. "CPU does not support the Supplemental SSE3 (SSSE3) instruction > set. This setup is unsupported and Impala might be unstable.") shoul > clarify this for other users. Should I submit a patch? > > That -march=native optimization might be useful especially for non-SSE4.2 > hardware -- our AMD CPUs support SSE2 and SSE4a, so there might be a > noticeable speedup for some workloads. Is there a way to switch that > optimization on or off? I would be curious about the speedup in our case. > > Best, > Tomáš > > On Thu, Jun 30, 2016 at 4:30 PM, Tim Armstrong <[email protected]> > wrote: > >> That's correct, historically when we added the warning there were some >> real issues related to SSSE3. We also didn't routinely test on hardware >> without that, so it was hard to be confident in it. >> >> We generate only 2 different SSE/non-SSE IR modules as you noticed, but I >> recently added the equivalent of -march=native for LLVM for optimization >> and code generation at runtime. The change made it into the Impala 2.6.0 >> release. I haven't seen any huge performance gains from that, but it >> definitely emits instructions up to and including AVX on my machine. If >> you're running an older version it won't do that. >> >> On Thu, Jun 30, 2016 at 1:48 AM, Tomáš Gavenčiak <[email protected]> >> wrote: >> >>> Thank you for the reply on packaging ... that was probably to be >>> expected. >>> >>> As for the instruction sets, our current servers (with AMD Opteron 4184 >>> from 2010) really do not support ssse3 and only support sse4a. I was going >>> to recompile Impala itself with compiler flags for these processors to gain >>> some speed, but it is true that this might not give us that much gain as >>> the query code is generated on fly via LLVM and that is where most of the >>> time is spent. >>> >>> A question for clarification, though: Since the code generated via LLVM >>> IR is either using SSE4.2 or no SSE at all, would it make sense to tweak >>> the -march flags of the query code generator? (e.g. set -march=native) I >>> haven't tried that, but might test it if you think this makes sense (e.g. >>> if the generated IR is not too low-level and already containing special >>> instructions or hints). >>> >>> Initially, we were quite confused by the "Exiting if Supplemental SSE3 >>> is not functional..." warning [1] marked as an error, but we got no >>> crashes (and CpuInfo::VerifyCpuRequirements() does not terminate the >>> program). If that is supposed to be a warning that there *might* be >>> crashes caused by lack of SSSE3, perhaps this could be clarified, or >>> removed. >>> >>> And big thanks for the great work on Impala! >>> >>> With best regards, >>> Tomáš >>> >>> [1] >>> https://github.com/cloudera/Impala/blob/2c78576a263208aa82dc9f0af73316eb2ea11eaf/be/src/util/cpu-info.cc#L138 >>> >>> >>> >>> >>> On Thu, Jun 30, 2016 at 6:59 AM, Tim Armstrong <[email protected]> >>> wrote: >>> >>>> I had a couple of questions, partially out of curiousity, and partially >>>> to understand what you're trying to do. >>>> >>>> What hardware are you running on? Last time we looked the most recent >>>> CPUs without ssse3 were AMD processors from around 2012. >>>> >>>> Which version of Impala are you running and are you actually running >>>> into crashes? I know we issue a warning at startup but I'm not actually >>>> sure that we currently use SSSE3 anywhere. We use various SSE4 instructions >>>> but those are all guarded with runtime checks. >>>> >>>> I'm also not actually sure what you would change about the build to >>>> solve the problem. We don't set -msse3 or anything like that on recent >>>> versions of Impala. >>>> >>>> >>>> On Wed, Jun 29, 2016 at 4:39 PM, Lars Volker <[email protected]> wrote: >>>> >>>>> Would it make sense to add a flag to impala to disable these features >>>>> during startup? >>>>> >>>>> On Wed, Jun 29, 2016 at 4:36 PM, Henry Robinson <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Tomáš - >>>>>> >>>>>> Unfortunately right now the packaging scripts are part of CDH, not >>>>>> Apache Impala itself, so they're not available (and might not even be >>>>>> very >>>>>> helpful, given that they probably have dependencies on Cloudera's >>>>>> infrastructure). >>>>>> >>>>>> It looks like copying binaries around might be the best thing for you >>>>>> to do for now. >>>>>> >>>>>> Thanks, >>>>>> Henry >>>>>> >>>>>> On 27 June 2016 at 09:34, Tomáš Gavenčiak <[email protected]> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I need to build the impala Ubuntu (.deb) packages from an already >>>>>>> built impala, and I cannot find any of the scripts. Where can I find >>>>>>> them? >>>>>>> >>>>>>> The reason is actually very simple but rather serious - the official >>>>>>> .deb packages rely on sse3, which is not supported on our (not even that >>>>>>> old) hardware. I managed to build an Impala version that runs well >>>>>>> (that is >>>>>>> the easy part), but I would like to avoid copying static (and possibly >>>>>>> outdated) binaries around my servers if possible. And being able to >>>>>>> build >>>>>>> my own .deb files would be a nice way around it! Note that the "deb-src" >>>>>>> repository does not list any sources for impala [1], and I haven't found >>>>>>> any such script in the (otherwise great!) impala-dev docker or the >>>>>>> impala >>>>>>> git repo either. >>>>>>> >>>>>>> (Another option would be to have an official build which would run >>>>>>> on older hardware, but that might be asking too much.) >>>>>>> >>>>>>> Thanks and regards! >>>>>>> Tomáš Gavenčiak, nic.cz >>>>>>> >>>>>>> [1] >>>>>>> http://archive.cloudera.com/impala/ubuntu/precise/amd64/impala/dists/precise-impala/contrib/source/ >>>>>>> and other vessions >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Impala Dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Henry Robinson >>>>>> Software Engineer >>>>>> Cloudera >>>>>> 415-994-6679 >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Impala Dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Impala Dev" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> >>>> >>> >> >
