[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-12-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648183#comment-17648183 ] Steve Loughran commented on HIVE-26699: --- in the builder pattern we use in hadoop. .opt() options

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17647671#comment-17647671 ] Steve Loughran commented on HIVE-26699: --- the api itself went in to hadoop earlier, in 3.3.0

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-11-12 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632676#comment-17632676 ] Steve Loughran commented on HIVE-26699: --- you should be using the openFile() api call and set the

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2022-10-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622163#comment-17622163 ] Steve Loughran commented on HIVE-16983: --- its fixed in hadoop-3.0+ with a moved to shaded AWS

[jira] [Commented] (HIVE-26063) Upgrade Apache parent POM to version 25

2022-10-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617189#comment-17617189 ] Steve Loughran commented on HIVE-26063: --- apparently this or an explicit update to the maven shade

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1 And Tez to 0.10.2

2022-08-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574635#comment-17574635 ] Steve Loughran commented on HIVE-24484: --- nice! > Upgrade Hadoop to 3.3.1 And Tez to 0.10.2 >

[jira] [Commented] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file

2022-06-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554217#comment-17554217 ] Steve Loughran commented on HIVE-25827: --- thanks. next question: do have one or more of * a

[jira] [Commented] (HIVE-25980) Reduce fs calls in HiveMetaStoreChecker.checkTable

2022-06-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553577#comment-17553577 ] Steve Loughran commented on HIVE-25980: --- ok. I'd still recommend the method {{listStatusIterator}}

[jira] [Commented] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file

2022-04-08 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519777#comment-17519777 ] Steve Loughran commented on HIVE-25827: --- is this per input stream, or are separate streams opened

[jira] [Commented] (HIVE-25980) Reduce fs calls in HiveMetaStoreChecker.checkTable

2022-03-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513422#comment-17513422 ] Steve Loughran commented on HIVE-25980: --- use listStatusIterator for incremental listing, page by

[jira] [Updated] (HIVE-25912) Drop external table at root of s3 bucket throws NPE

2022-02-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-25912: -- Summary: Drop external table at root of s3 bucket throws NPE (was: Drop external table throw

[jira] [Commented] (HIVE-24852) Add support for Snapshots during external table replication

2021-11-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436757#comment-17436757 ] Steve Loughran commented on HIVE-24852: --- # Does this downgrade properly when the destination FS is

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1

2021-09-09 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412556#comment-17412556 ] Steve Loughran commented on HIVE-24484: --- HADOOP-17313 actually went in to deal with hive processes

[jira] [Commented] (HIVE-24546) Avoid unwanted cloud storage call during dynamic partition load

2021-07-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380090#comment-17380090 ] Steve Loughran commented on HIVE-24546: --- I'd recommend * skip the dest path check * call mkdirs()

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372145#comment-17372145 ] Steve Loughran commented on HIVE-24849: --- Something like this * existence check is integrated

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371535#comment-17371535 ] Steve Loughran commented on HIVE-24849: --- How does tbl.isEmpty() work? Does it do a listStatus call

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1

2021-06-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368073#comment-17368073 ] Steve Loughran commented on HIVE-24484: --- bq. Would be great if folks could work on syncing the

[jira] [Commented] (HIVE-24916) EXPORT TABLE command to ADLS Gen2/s3 failing

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367447#comment-17367447 ] Steve Loughran commented on HIVE-24916: --- If the hadoop version is recent, then calling

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367439#comment-17367439 ] Steve Loughran commented on HIVE-24849: --- [~glapark] bq. Now, HiveServer2 does not send

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367438#comment-17367438 ] Steve Loughran commented on HIVE-24849: --- is hive doing its own recursive treewalk or calling

[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367301#comment-17367301 ] Steve Loughran commented on HIVE-17133: --- Is this ready to go in? even without a new test? >

[jira] [Commented] (HIVE-24717) Migrate to listStatusIterator in moving files

2021-02-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277866#comment-17277866 ] Steve Loughran commented on HIVE-24717: --- happy to review a hadoop PR with the relevant fix

[jira] [Updated] (HIVE-23492) Remove unnecessary FileSystem#exists calls from ql module

2020-07-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-23492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-23492: -- Description: Wherever there is an exists() call before open() or delete(), remove it and infer

[jira] [Commented] (HIVE-22819) Refactor Hive::listFilesCreatedByQuery to make it faster for object stores

2020-02-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044471#comment-17044471 ] Steve Loughran commented on HIVE-22819: --- LGTM -this saves two round trips to HDFS, S3 or ABFS. >

[jira] [Commented] (HIVE-14165) Remove Hive file listing during split computation

2020-02-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033607#comment-17033607 ] Steve Loughran commented on HIVE-14165: --- What is the current status of this? Is it a defacto

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2020-01-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007496#comment-17007496 ] Steve Loughran commented on HIVE-16295: --- yeah, where are we with this? Is anyone active on it? >

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-12-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989710#comment-16989710 ] Steve Loughran commented on HIVE-22548: --- OK. BTW, if you call toString on the S3A connector you

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987093#comment-16987093 ] Steve Loughran commented on HIVE-22548: --- do you need that return code from

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-11-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983416#comment-16983416 ] Steve Loughran commented on HIVE-22548: --- Also L1644 it calls path.exists() before the listFiles.

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964170#comment-16964170 ] Steve Loughran commented on HIVE-22411: --- patch looks functional to me at a glance There is still a

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962280#comment-16962280 ] Steve Loughran commented on HIVE-22411: --- FYI [~gabor.bota][~rajesh.balamohan] > Performance

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962271#comment-16962271 ] Steve Loughran commented on HIVE-22411: --- Why do you need to list every single file under a

[jira] [Commented] (HIVE-22054) Avoid recursive listing to check if a directory is empty

2019-07-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896009#comment-16896009 ] Steve Loughran commented on HIVE-22054: --- you are correct, the getContentSummary call will be

[jira] [Resolved] (HIVE-19580) Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR

2019-02-20 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HIVE-19580. --- Resolution: Not A Problem OK. closing. Trying hard to think of the best way to classify,

[jira] [Updated] (HIVE-19580) Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR

2019-02-20 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-19580: -- Summary: Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR (was: Hive 2.3.2

[jira] [Comment Edited] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772255#comment-16772255 ] Steve Loughran edited comment on HIVE-19580 at 2/19/19 7:21 PM: If this

[jira] [Commented] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772255#comment-16772255 ] Steve Loughran commented on HIVE-19580: --- If this is EMR then AWS are the only person who can deal

[jira] [Updated] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-19580: -- Environment: EMR s3:// connector Spark 2.3 but also true for lower versions Hive 2.3.2

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2018-11-02 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672912#comment-16672912 ] Steve Loughran commented on HIVE-16913: --- DTs aren't sufficient here as Hive uses its granted

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-07-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554709#comment-16554709 ] Steve Loughran commented on HIVE-16295: --- w.r.t maven dependencies, if you are building against

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506017#comment-16506017 ] Steve Loughran commented on HIVE-16391: --- I'm pleased to see the kryo version stuff isn't an issue

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-06-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503216#comment-16503216 ] Steve Loughran commented on HIVE-16295: --- * PathOutputCommitterFactory; you can ask for that to

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502129#comment-16502129 ] Steve Loughran commented on HIVE-16391: --- bq. The problem with that is that it changes the meaning

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501534#comment-16501534 ] Steve Loughran commented on HIVE-16391: --- Generally uses .patch files attached to the JIRA >

[jira] [Commented] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2018-05-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495447#comment-16495447 ] Steve Loughran commented on HIVE-19580: --- Don't see why this should be s3-related. * Can you

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452645#comment-16452645 ] Steve Loughran commented on HIVE-16295: --- bq. is there a reason PathOutputCommitterFactory doesn't

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452112#comment-16452112 ] Steve Loughran commented on HIVE-16295: --- One other comment: you can rely on _SUCCESS being a JSON

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450480#comment-16450480 ] Steve Loughran commented on HIVE-16295: --- Impressive. I'm not knowledgeable about hive to review this

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392751#comment-16392751 ] Steve Loughran commented on HIVE-18861: --- thx for your help nurturing this in. > druid-hdfs-storage

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391555#comment-16391555 ] Steve Loughran commented on HIVE-18861: --- I don't see these tests being related' there's nothing

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK,

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390366#comment-16390366 ] Steve Loughran commented on HIVE-18861: --- Not seeing any updates after 9h. Cancelling and reattaching

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) got it; cut the -version marker. You must be using a

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK,

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861-001.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387697#comment-16387697 ] Steve Loughran commented on HIVE-18861: --- [~ashutoshc]: I dont see jira running tests here...is there

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386345#comment-16386345 ] Steve Loughran commented on HIVE-18861: --- thanks! If this goes it in, it will be first contribution

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386228#comment-16386228 ] Steve Loughran commented on HIVE-18861: --- Dependencies before the patch when built against hadoop

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386231#comment-16386231 ] Steve Loughran commented on HIVE-18861: --- And after {code} [INFO] | +-

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861-001.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386225#comment-16386225 ] Steve Loughran commented on HIVE-18861: --- Patch 001: pulls the hadoop JAR and the aws sdk version,

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Summary: druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Description: druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Summary: druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on

[jira] [Assigned] (HIVE-18861) druid-server is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HIVE-18861: - > druid-server is pulling in hadoop-aws-2.7.2, creating classpath problems on > hadoop 3.x

[jira] [Commented] (HIVE-1620) Patch to write directly to S3 from Hive

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386192#comment-16386192 ] Steve Loughran commented on HIVE-1620: -- This is the wrong way to handle variations in FS semantics;

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082117#comment-16082117 ] Steve Loughran commented on HIVE-16983: --- * The joda time update will be mandatory for S3A to auth *

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080062#comment-16080062 ] Steve Loughran commented on HIVE-16983: --- Patch itself LGTM from an S3a perspective one thing for

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072760#comment-16072760 ] Steve Loughran commented on HIVE-16983: --- good point Everyone: look at the S3A troubleshooting docs

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072239#comment-16072239 ] Steve Loughran commented on HIVE-16983: --- Clearly, somehow, your credentials aren't getting picked

[jira] [Commented] (HIVE-9012) Not able to move and populate the data fully on to the table when the scratch directory is on S3

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072238#comment-16072238 ] Steve Loughran commented on HIVE-9012: -- This is just rename() being emulated in S3 with a

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-29 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068127#comment-16068127 ] Steve Loughran commented on HIVE-16913: --- You are going to need a multi-tenant Hive service, such as

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066255#comment-16066255 ] Steve Loughran commented on HIVE-16913: --- Note that if you try and be clever about key names, then

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066253#comment-16066253 ] Steve Loughran commented on HIVE-16913: --- # credentials on Hadoop 2.7+ can go in JCEKs files too.

[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t

2017-05-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017265#comment-16017265 ] Steve Loughran commented on HIVE-16446: --- # try with s3a URS and the fs.s3a secret and access keys #

[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t

2017-04-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979869#comment-15979869 ] Steve Loughran commented on HIVE-16446: --- you should switch to using s3a:// URLs in things based on

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's OutputCommitter

2017-04-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965660#comment-15965660 ] Steve Loughran commented on HIVE-16295: --- Thanks for starting this 1. We're making changes to

[jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory

2017-03-04 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895818#comment-15895818 ] Steve Loughran commented on HIVE-14864: --- {{FileSystem.getContentSummary()}} does a recursive

[jira] [Commented] (HIVE-15502) CTAS on S3 is broken with credentials exception

2017-03-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890221#comment-15890221 ] Steve Loughran commented on HIVE-15502: --- probably comes down to the ordering of the FS creation vs

[jira] [Commented] (HIVE-15368) consider optimizing Utilities::handleMmTableFinalPath

2017-03-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890216#comment-15890216 ] Steve Loughran commented on HIVE-15368: --- If you can use {{FileSystem.listFiles(path,

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-29 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785633#comment-15785633 ] Steve Loughran commented on HIVE-15016: --- don't think Hadoop is making much use of codahale or

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756877#comment-15756877 ] Steve Loughran commented on HIVE-15016: --- if you check out hadoop trunk, all you need to do is make a

[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT

2016-12-02 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714928#comment-15714928 ] Steve Loughran commented on HIVE-15326: --- HIVE-15016 includes a fix for that, simply by changing the

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712671#comment-15712671 ] Steve Loughran commented on HIVE-15016: --- What's the issue with the codahale JAR? Incompatible with

[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT

2016-12-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711687#comment-15711687 ] Steve Loughran commented on HIVE-15326: --- Test is easy; attempt to instantiate a HiveConf {code}

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687108#comment-15687108 ] Steve Loughran commented on HIVE-15199: --- I do think I'd rather fix this in s3, because it is adding

[jira] [Comment Edited] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684283#comment-15684283 ] Steve Loughran edited comment on HIVE-15199 at 11/22/16 3:19 PM: - you are

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-21 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684283#comment-15684283 ] Steve Loughran commented on HIVE-15199: --- you are right, I am wrong: serves me right for commenting

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679190#comment-15679190 ] Steve Loughran commented on HIVE-15199: --- if you do listStatus(path, recursive=true) you don't get

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673317#comment-15673317 ] Steve Loughran commented on HIVE-15199: --- # as sahil notes, blobstore copy calls must be in object

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667931#comment-15667931 ] Steve Loughran commented on HIVE-15199: --- sounds related to HADOOP-13402 I am not going to express

[jira] [Comment Edited] (HIVE-15093) S3-to-S3 Renames: Files should be moved individually rather than at a directory level

2016-11-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653694#comment-15653694 ] Steve Loughran edited comment on HIVE-15093 at 11/10/16 10:32 AM: -- # I've

[jira] [Commented] (HIVE-15093) S3-to-S3 Renames: Files should be moved individually rather than at a directory level

2016-11-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653694#comment-15653694 ] Steve Loughran commented on HIVE-15093: --- #. I've just started HADOOP-13600, though busy with

[jira] [Commented] (HIVE-15093) S3-to-S3 Renames: Files should be moved individually rather than at a directory level

2016-11-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651394#comment-15651394 ] Steve Loughran commented on HIVE-15093: --- -1 (non binding) Doing parallel rename is a stop-gap

  1   2   >