[ https://issues.apache.org/jira/browse/IMPALA-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365195#comment-17365195 ]
ASF subversion and git services commented on IMPALA-9690: --------------------------------------------------------- Commit ab3640edf1d4e4c492187fb3a9f6a07e0f4ab9ba in impala's branch refs/heads/branch-4.0.0 from Joe McDonnell [ https://gitbox.apache.org/repos/asf?p=impala.git;h=ab3640e ] IMPALA-9690 addendum: Add a legacy AVX mode Impala 4 introduced a requirement that x86_64 machines support AVX2. This adds a startup parameter enable_legacy_avx_support that relaxes the AVX2 requirement to require only AVX. By default, enable_legacy_avx_support is false. enable_legacy_avx_support will be removed in a future release, but it seems like a sensible way to support older machines for now. Testing: - Ran with enable_legacy_avx_support on a pool of machines that includes machines without AVX2 - Ran a core job on a machine with AVX2 Change-Id: I44051720697a2a497f9017e4b31289be77fb6b05 Reviewed-on: http://gerrit.cloudera.org:8080/17588 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Joe McDonnell <joemcdonn...@cloudera.com> Reviewed-by: Daniel Becker <daniel.bec...@cloudera.com> Reviewed-by: Quanlong Huang <huangquanl...@gmail.com> > Bump minimum x86-64 CPU requirements > ------------------------------------ > > Key: IMPALA-9690 > URL: https://issues.apache.org/jira/browse/IMPALA-9690 > Project: IMPALA > Issue Type: Improvement > Components: Backend > Reporter: Tim Armstrong > Assignee: Joe McDonnell > Priority: Blocker > Labels: incompatibility, performance > Fix For: Impala 4.0 > > > We still have a minimum CPU requirement of SSSE3 support > https://impala.apache.org/docs/build/html/topics/impala_prereqs.html. I.e. we > don't assume SSE4.2 or AVX or AVX2. > There is a lot of legacy code to support CPUs without SSE4.2 and various > other extensions. As a start, here are all the locations in the code where we > branch based on CPU feature: > {noformat} > :~/impala/impala$ git grep CpuInfo::IsSupport > be/src/benchmarks/int-hash-benchmark.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_2)) suite32.BENCH(uint32_t, CRC); > be/src/benchmarks/int-hash-benchmark.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_2)) { > be/src/benchmarks/int-hash-benchmark.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_1)) { > be/src/benchmarks/int-hash-benchmark.cc: if > (CpuInfo::IsSupported(CpuInfo::AVX2)) { > be/src/benchmarks/string-compare-benchmark.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_2)) { > be/src/exec/delimited-text-parser.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_2)) { > be/src/exec/delimited-text-parser.cc: if > (CpuInfo::IsSupported(CpuInfo::SSE4_2)) { > be/src/exec/delimited-text-parser.inline.h: > DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/exec/delimited-text-parser.inline.h: if > (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) { > be/src/runtime/io/disk-io-mgr.cc: if > (!CpuInfo::IsSupported(CpuInfo::SSE4_2)) { > be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::SSSE3)) { > be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) { > be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(cpu_info_flag)) { > be/src/util/bit-util-test.cc:// CpuInfo::IsSupported() checks. This doesn't > test the bug precisely but is a canary for > be/src/util/bit-util.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) { > be/src/util/bit-util.cc: } else if > (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) { > be/src/util/bit-util.cc: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) > { > be/src/util/bit-util.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::POPCNT))) > { > be/src/util/bloom-filter.cc: if (CpuInfo::IsSupported(CpuInfo::AVX)) { > be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) { > be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) { > be/src/util/cpu-info.cc: if (!CpuInfo::IsSupported(CpuInfo::SSSE3)) { > be/src/util/cpu-info.h: /// // line, CpuInfo::IsSupported(CpuInfo::AVX2) > will return false. > be/src/util/cpu-info.h: : feature_(feature), > reenable_(CpuInfo::IsSupported(feature)) { > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2)); > be/src/util/hash-util.h: if > (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) { > be/src/util/openssl-util.cc: return > (CpuInfo::IsSupported(CpuInfo::PCLMULQDQ) > {noformat} > We also ship two versions of the codegen module, one of which (nosse42) is > essentially never used. > I think it would be uncontroversial to bump the minimum requirement to > SSE4.2, which would allow us to delete some old fallbacks. I think the last > time Intel or AMD shipped a processor without this was 2010 or 2011. Jumping > to AVX is probably almost as uncontroversial, since it looks like that has > been universal for nearly as long: > https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX. > Some older lower-performance cloud instance types don't support AVX, it looks > like, but I think this is an edge case. > It would be very nice to require AVX2 because that could remove a bunch of > conditional code (I think the contributors adding ARM support might want to > keep the scalar fallback code though, potentially). It looks like most Intel > and AMD processors have supported it since 2013 and 2015 respectively, except > some low-end Intel processors: > https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2. > In the past we've been a bit conservative about this - people do sometimes > use very old hardware as a test cluster because it was the hardware that they > could get their hands on (e.g. I remember someone complaining several years > back that they couldn't get Impala running on some old AMD CPUs that were > missing SSSE3 support). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org