[jira] [Created] (HIVE-11439) PERCENTILE_APPROX is inconsistent with PERCENTILE even for a small number of data points
Mikhail Bautin created HIVE-11439: - Summary: PERCENTILE_APPROX is inconsistent with PERCENTILE even for a small number of data points Key: HIVE-11439 URL: https://issues.apache.org/jira/browse/HIVE-11439 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[ https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-4525: - Status: Patch Available (was: Open) Support timestamps earlier than 1970 and later than 2038 Key: HIVE-4525 URL: https://issues.apache.org/jira/browse/HIVE-4525 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D10755.1.patch, D10755.2.patch TimestampWritable currently serializes timestamps using the lower 31 bits of an int. This does not allow to store timestamps earlier than 1970 or later than a certain point in 2038. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[ https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661870#comment-13661870 ] Mikhail Bautin commented on HIVE-4525: -- Test results with and without this patch differ only by a spurious failure of a ZK-related test that is not affected by the changes here. *** 3838,3843 --- 3838,3845 [junit] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 [junit] Running org.apache.hadoop.hive.serde2.dynamic_type.TestDynamicSerDe [junit] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0 + [junit] Running org.apache.hadoop.hive.serde2.io.TestTimestampWritable + [junit] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 [junit] Running org.apache.hadoop.hive.serde2.lazy.TestLazyArrayMapStruct [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [junit] Running org.apache.hadoop.hive.serde2.lazy.TestLazyPrimitive *** *** 3901,3906 [junit] Running org.apache.hcatalog.hbase.snapshot.TestZNodeSetUp [junit] Tests run: 0, Failures: 0, Errors: 2, Skipped: 0 [junit] Running org.apache.hcatalog.hbase.snapshot.lock.WriteLockTest ! [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [junit] Running org.apache.hcatalog.hbase.snapshot.lock.ZNodeNameTest [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 --- 3903,3908 [junit] Running org.apache.hcatalog.hbase.snapshot.TestZNodeSetUp [junit] Tests run: 0, Failures: 0, Errors: 2, Skipped: 0 [junit] Running org.apache.hcatalog.hbase.snapshot.lock.WriteLockTest ! [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 [junit] Running org.apache.hcatalog.hbase.snapshot.lock.ZNodeNameTest [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 + set +x Committers: could you please take a look and consider committing this? Cc [~ashutoshc], [~owen.omalley], [~cwsteinbach]. Thanks! Support timestamps earlier than 1970 and later than 2038 Key: HIVE-4525 URL: https://issues.apache.org/jira/browse/HIVE-4525 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D10755.1.patch, D10755.2.patch TimestampWritable currently serializes timestamps using the lower 31 bits of an int. This does not allow to store timestamps earlier than 1970 or later than a certain point in 2038. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[
https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659128#comment-13659128
]
Mikhail Bautin commented on HIVE-4525:
--
I am not quite sure how to solve the backward compatibility issue in the
writable part of {{TimestampWritable}} code ({{write}}/{{readFields}}) by
switching to a unified nanosecond-timestamp-as-long format. If {{readFields}}
is presented with eight bytes, would it interpret them as a four-byte int
followed by a VInt or as a long nanosecond timestamp? Would it attempt to do
the former and revert to the latter if there are inconsistencies? What if the
bytes of a long nanosecond timestamp also happen to represent a valid legacy
(int/VInt) timestamp?
In my patch, I try to maintain backward compatibility as much as possible. If a
timestamp is in the range that can be represented by the old format, it is
serialized using the old format. The extended format I've proposed and
implemented for the full timestamp range builds on top of the existing one and
can be unambiguously distinguished from the old format by examining serialized
bytes.
In addition, the included test, {{TestTimestampWritable}}, tests both the old
and the new (extended format), as well as double/BigDecimal conversion,
getters/setters/constructors and everything else I could test in
{{TimestampWritable}}.
I am sure there is a way to handle vector optimizations for timestamps in a
backward-compatible way, and I don't think this patch would make it much more
complicated than it already is. However, vectorized computations are a
performance optimization, while this issue is a correctness fix. Currently,
timestamps outside of the ~1970-2038 range would be silently corrupted in some
queries, and this patch successfully fixes that. It is also pretty small and
immediately available.
Support timestamps earlier than 1970 and later than 2038
Key: HIVE-4525
URL: https://issues.apache.org/jira/browse/HIVE-4525
Project: Hive
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D10755.1.patch
TimestampWritable currently serializes timestamps using the lower 31 bits of
an int. This does not allow to store timestamps earlier than 1970 or later
than a certain point in 2038.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[
https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656141#comment-13656141
]
Mikhail Bautin commented on HIVE-4525:
--
Correction to the design of this feature (I can't edit comments because of
permissions, so adding another comment). In case the seconds field needs more
than 31 bit, the first VInt is {{-1-reversedDecimal}} regardless of whether
{{reversedDecimal}} is zero or not.
Support timestamps earlier than 1970 and later than 2038
Key: HIVE-4525
URL: https://issues.apache.org/jira/browse/HIVE-4525
Project: Hive
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D10755.1.patch
TimestampWritable currently serializes timestamps using the lower 31 bits of
an int. This does not allow to store timestamps earlier than 1970 or later
than a certain point in 2038.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[ https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656271#comment-13656271 ] Mikhail Bautin commented on HIVE-4525: -- [~ehans]: switching to long nanosecond timestamps would definitely be a much nicer solution, but don't you think it would break backward-compatibility for timestamps serialized using the old format? Support timestamps earlier than 1970 and later than 2038 Key: HIVE-4525 URL: https://issues.apache.org/jira/browse/HIVE-4525 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D10755.1.patch TimestampWritable currently serializes timestamps using the lower 31 bits of an int. This does not allow to store timestamps earlier than 1970 or later than a certain point in 2038. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4220) TimestampWritable.toString throws array index exception sometimes
[
https://issues.apache.org/jira/browse/HIVE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652133#comment-13652133
]
Mikhail Bautin commented on HIVE-4220:
--
[~navis]: I think HIVE-4516 solves the same problem but in somewhat simpler way
(using ThreadLocal). Could you please take a look at the patch there and let us
know what you think?
[~ashutoshc]: I think it is reasonable to assume at this point that Hive
primitives, especially as low-level as TimestampWritable, have to be
thread-safe. This is not only required by third-party low-latency query
processing systems such as AmpLab's Shark, but also by the effort in the Hive
community itself to speed up query processing (e.g.
http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/) that I
believe will inevitably require keeping pre-existing multi-threaded executor
JVMs around.
TimestampWritable.toString throws array index exception sometimes
-
Key: HIVE-4220
URL: https://issues.apache.org/jira/browse/HIVE-4220
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4220.D9669.1.patch
{noformat}
org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
java.lang.ArrayIndexOutOfBoundsException: 45
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:215)
at
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170)
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:288)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:348)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:194)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1449)
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:193)
... 11 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 45
at
sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
at
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
at
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996)
at java.util.Calendar.setTimeInMillis(Calendar.java:1110)
at java.util.Calendar.setTime(Calendar.java:1076)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:875)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:868)
at java.text.DateFormat.format(DateFormat.java:316)
at
org.apache.hadoop.hive.serde2.io.TimestampWritable.toString(TimestampWritable.java:327)
at
org.apache.hadoop.hive.serde2.lazy.LazyTimestamp.writeUTF8(LazyTimestamp.java:95)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:234)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:381)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365)
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:96)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
at
[jira] [Created] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
Mikhail Bautin created HIVE-4525: Summary: Support timestamps earlier than 1970 and later than 2038 Key: HIVE-4525 URL: https://issues.apache.org/jira/browse/HIVE-4525 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin TimestampWritable currently serializes timestamps using the lower 31 bits of an int. This does not allow to store timestamps earlier than 1970 or later than a certain point in 2038. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[
https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652676#comment-13652676
]
Mikhail Bautin commented on HIVE-4525:
--
h4. Design proposal
We have to be able to read the current {{TimestampWritable}}-serializable
format for backward-compatibility, and write the format recognizable by the
current {{TimestampWritable}} implementation for timestamps within the
currently supported range. We can use the negative range of the {{VInt}} in the
binary representation of the timestamp that normally represents the reversed
decimal part to indicate the presence of an additional {{VInt}} field that
stores the remaining bits of the {{seconds}} number (i.e. {{seconds 31}}).
The meaning of the 7th bit of the first byte then changes from has decimal to
has decimal or 31 bits of seconds.
The following table summarizes the four logical cases of timestamp
serialization. The first two are backward-compatible. The second two cases are
unsupported by the current format, so they will not be recognized by the
current version.
|| Seconds need 31 bits || Has decimal || 7th bit of the first byte || First
VInt || Second VInt ||
| No | No | {{0}} | N/A | N/A |
| No | Yes | {{1}} | {{reversedDecimal}} | N/A |
| Yes | No | {{1}} | {{-1}} | {{seconds 31}} |
| Yes | Yes | {{1}} | {{-2 - reversedDecimal}} | {{seconds 31}} |
Support timestamps earlier than 1970 and later than 2038
Key: HIVE-4525
URL: https://issues.apache.org/jira/browse/HIVE-4525
Project: Hive
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
TimestampWritable currently serializes timestamps using the lower 31 bits of
an int. This does not allow to store timestamps earlier than 1970 or later
than a certain point in 2038.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4525) Support timestamps earlier than 1970 and later than 2038
[ https://issues.apache.org/jira/browse/HIVE-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652709#comment-13652709 ] Mikhail Bautin commented on HIVE-4525: -- Also, the binary-sortable representation of timestamps would have to change to accommodate additional high-order bits. If a 4-byte second-precision timestamp covers 68 years (or 136 if signed), by adding one most-significant byte we can cover 17408 (or 34816) years, which is good enough for all practical purposes. Support timestamps earlier than 1970 and later than 2038 Key: HIVE-4525 URL: https://issues.apache.org/jira/browse/HIVE-4525 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin TimestampWritable currently serializes timestamps using the lower 31 bits of an int. This does not allow to store timestamps earlier than 1970 or later than a certain point in 2038. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3930) Generate and publish source jars
[ https://issues.apache.org/jira/browse/HIVE-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601576#comment-13601576 ] Mikhail Bautin commented on HIVE-3930: -- There is a patch for a the AMP Lab version of Hive 0.9: https://github.com/amplab/hive/pull/10/files Generate and publish source jars Key: HIVE-3930 URL: https://issues.apache.org/jira/browse/HIVE-3930 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Hive should generate and publish source jars to Maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3974) RCFile needs to export a method to flush buffered records and provide a flush callback
Mikhail Bautin created HIVE-3974: Summary: RCFile needs to export a method to flush buffered records and provide a flush callback Key: HIVE-3974 URL: https://issues.apache.org/jira/browse/HIVE-3974 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Sometimes it is desirable to make sure all buffered records have been persisted, and take some action when that happens. The easiest way to achieve this is to make the flushRecords() method public and add a callback for when a flush happens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3930) Generate and publish source jars
Mikhail Bautin created HIVE-3930: Summary: Generate and publish source jars Key: HIVE-3930 URL: https://issues.apache.org/jira/browse/HIVE-3930 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Hive should generate and publish source jars to Maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3930) Generate and publish source jars
[ https://issues.apache.org/jira/browse/HIVE-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561386#comment-13561386 ] Mikhail Bautin commented on HIVE-3930: -- [~ashutoshc]: HIVE-3830 seems to talk about generating a source-only release (a tarball, as far as I understand), while here I am talking about generating source jars that could be distributed through Maven to make it easier to develop projects that depend on Hive. Generate and publish source jars Key: HIVE-3930 URL: https://issues.apache.org/jira/browse/HIVE-3930 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Hive should generate and publish source jars to Maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3913) Possible deadlock in ZK lock manager
[ https://issues.apache.org/jira/browse/HIVE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13559830#comment-13559830 ] Mikhail Bautin commented on HIVE-3913: -- [~ashutoshc]: thanks for pointing this out. I took a look at HIVE-3537, and I think this is a much simpler fix for a situation that I observed when the client attempts to acquire a lock, but the ZK connection is closed for some reason and no reconnection is attempted. Possible deadlock in ZK lock manager Key: HIVE-3913 URL: https://issues.apache.org/jira/browse/HIVE-3913 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Attachments: D8097.1.patch ZK Hive lock manager can get into a state when the connection is closed, but no reconnection is attempted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3036) hive should support BigDecimal datatype
[ https://issues.apache.org/jira/browse/HIVE-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13557476#comment-13557476 ] Mikhail Bautin commented on HIVE-3036: -- Actually this seems to be a duplicate of HIVE-2693. hive should support BigDecimal datatype --- Key: HIVE-3036 URL: https://issues.apache.org/jira/browse/HIVE-3036 Project: Hive Issue Type: New Feature Components: Query Processor, Types Affects Versions: 0.7.1, 0.8.0, 0.8.1 Reporter: Anurag Tangri hive has support for big int but people have use cases where they need decimal precision to a big value. Values in question are like decimal(x,y). for eg. decimal of form (17,6) which cannot be represented by float/double. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3913) Possible deadlock in ZK lock manager
Mikhail Bautin created HIVE-3913: Summary: Possible deadlock in ZK lock manager Key: HIVE-3913 URL: https://issues.apache.org/jira/browse/HIVE-3913 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical ZK Hive lock manager can get into a state when the connection is closed, but no reconnection is attempted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3854) Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin)
Mikhail Bautin created HIVE-3854: Summary: Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin) Key: HIVE-3854 URL: https://issues.apache.org/jira/browse/HIVE-3854 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: 0001-Added-a-condition-in-input_part8-to-make-the-result-.patch Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin). Needed for the Shark project. https://github.com/amplab/hive/commit/3e7606bb4793e000e7a0e51b8565bf1bec2dbf33 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3854) Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin)
[ https://issues.apache.org/jira/browse/HIVE-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3854: - Attachment: 0001-Added-a-condition-in-input_part8-to-make-the-result-.patch Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin) -- Key: HIVE-3854 URL: https://issues.apache.org/jira/browse/HIVE-3854 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: 0001-Added-a-condition-in-input_part8-to-make-the-result-.patch Add a condition in input_part8 to make the result deterministic (patch by Reynold Xin). Needed for the Shark project. https://github.com/amplab/hive/commit/3e7606bb4793e000e7a0e51b8565bf1bec2dbf33 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3855) Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin)
Mikhail Bautin created HIVE-3855: Summary: Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin) Key: HIVE-3855 URL: https://issues.apache.org/jira/browse/HIVE-3855 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin). Needed by the Shark project (http://shark.cs.berkeley.edu/). https://github.com/amplab/hive/commit/31be6d8a61f454293f13e0d8bb2d88c3dce19c44 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3855) Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin)
[ https://issues.apache.org/jira/browse/HIVE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3855: - Attachment: Fixed-hive-tests-with-non-deterministic-output-e.g.-.patch Patch for branch-0.9. Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin) --- Key: HIVE-3855 URL: https://issues.apache.org/jira/browse/HIVE-3855 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: Fixed-hive-tests-with-non-deterministic-output-e.g.-.patch Fix tests with non-deterministic output, e.g. limit without order by (patch by Reynold Xin). Needed by the Shark project (http://shark.cs.berkeley.edu/). https://github.com/amplab/hive/commit/31be6d8a61f454293f13e0d8bb2d88c3dce19c44 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3809) Concurrency issue in RCFile: multiple threads can use the same decompressor
[ https://issues.apache.org/jira/browse/HIVE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3809: - Attachment: 0001-HIVE-3809-Concurrency-issue-in-RCFile-multiple-threa.patch Concurrency issue in RCFile: multiple threads can use the same decompressor --- Key: HIVE-3809 URL: https://issues.apache.org/jira/browse/HIVE-3809 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.11.0 Attachments: 0001-HIVE-3809-Concurrency-issue-in-RCFile-multiple-threa.patch, 0001-HIVE-3809-Decompressors-should-only-be-returned-to-t.patch, D7419.1.patch RCFile is not thread-safe, even if each reader is only used by one thread as intended, because it is possible to return decompressors to the pool multiple times by calling close on the reader multiple times. Then, different threads can pick up the same decompressor twice from the pool, resulting in decompression failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3809) Concurrency issue in RCFile: multiple threads can use the same decompressor
[ https://issues.apache.org/jira/browse/HIVE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540660#comment-13540660 ] Mikhail Bautin commented on HIVE-3809: -- [~ashutoshc]: thank you for committing this to trunk. Could you please also commit this to branch 0.9? Patch for branch 0.9: https://issues.apache.org/jira/secure/attachment/12562623/0001-HIVE-3809-Concurrency-issue-in-RCFile-multiple-threa.patch Concurrency issue in RCFile: multiple threads can use the same decompressor --- Key: HIVE-3809 URL: https://issues.apache.org/jira/browse/HIVE-3809 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical Fix For: 0.11.0 Attachments: 0001-HIVE-3809-Concurrency-issue-in-RCFile-multiple-threa.patch, 0001-HIVE-3809-Decompressors-should-only-be-returned-to-t.patch, D7419.1.patch RCFile is not thread-safe, even if each reader is only used by one thread as intended, because it is possible to return decompressors to the pool multiple times by calling close on the reader multiple times. Then, different threads can pick up the same decompressor twice from the pool, resulting in decompression failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[
https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534210#comment-13534210
]
Mikhail Bautin commented on HIVE-3645:
--
I am getting the following compilation errors with this patch on branch-0.9:
{code}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 1 source file to /wd/hive/build/shims/classes
[javac]
/wd/hive/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:118:
error: method getDefaultBlockSize in class FileSystem cannot be applied to
given types;
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac] required: no arguments
[javac] found: Path
[javac] reason: actual and formal argument lists differ in length
[javac]
/wd/hive/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:123:
error: method getDefaultReplication in class FileSystem cannot be applied to
given types;
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] required: no arguments
[javac] found: Path
[javac] reason: actual and formal argument lists differ in length
[javac] 2 errors
BUILD FAILED
/wd/hive/build.xml:319: The following error occurred while executing this line:
/wd/hive/build.xml:169: The following error occurred while executing this line:
/wd/hive/shims/build.xml:90: The following error occurred while executing this
line:
/wd/hive/shims/build.xml:93: The following error occurred while executing this
line:
/wd/hive/shims/build.xml:82: Compile failed; see the compiler error output for
details.
{code}
I am building with the default Hadoop version.
RCFileWriter does not implement the right function to support Federation
Key: HIVE-3645
URL: https://issues.apache.org/jira/browse/HIVE-3645
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.9.0, 0.10.0
Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10
Reporter: Viraj Bhat
Assignee: Arup Malakar
Fix For: 0.11
Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch
Create a table using Hive DDL
{code}
CREATE TABLE tmp_hcat_federated_numbers_part_1 (
id int,
intnum int,
floatnum float
)partitioned by (
part1string,
part2string
)
STORED AS rcfile
LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1';
{code}
Populate it using Pig:
{code}
A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader();
B = filter A by id = 500;
C = foreach B generate (int)id, (int)intnum, (float)floatnum;
store C into
'default.tmp_hcat_federated_numbers_part_1'
using org.apache.hcatalog.pig.HCatStorer
('part1=pig, part2=hcat_pig_insert',
'id: int,intnum: int,floatnum: float');
{code}
Generates the following error when running on a Federated Cluster:
{quote}
2012-10-29 20:40:25,011 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3
Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException:
getDefaultReplication on empty path is invalid
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705)
at
org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86)
at
org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100)
at
org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more
[jira] [Created] (HIVE-3813) Allow publishing artifacts to an arbitrary remote repository
Mikhail Bautin created HIVE-3813: Summary: Allow publishing artifacts to an arbitrary remote repository Key: HIVE-3813 URL: https://issues.apache.org/jira/browse/HIVE-3813 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Allow publishing artifacts to an arbitrary remote repository by specifying -Dmvn.publish.repoUrl on the command line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3813) Allow publishing artifacts to an arbitrary remote repository
[ https://issues.apache.org/jira/browse/HIVE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3813: - Description: Allow publishing artifacts to an arbitrary remote repository by specifying -Dmvn.publish.repoUrl on the command line (patch by Thomas Dudziak). (was: Allow publishing artifacts to an arbitrary remote repository by specifying -Dmvn.publish.repoUrl on the command line.) Allow publishing artifacts to an arbitrary remote repository Key: HIVE-3813 URL: https://issues.apache.org/jira/browse/HIVE-3813 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Allow publishing artifacts to an arbitrary remote repository by specifying -Dmvn.publish.repoUrl on the command line (patch by Thomas Dudziak). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3813) Allow publishing artifacts to an arbitrary remote repository
[ https://issues.apache.org/jira/browse/HIVE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3813: - Attachment: 0001-HIVE-3813-Allow-publishing-artifacts-to-an-arbitrary.patch Attaching a manually generated patch. Allow publishing artifacts to an arbitrary remote repository Key: HIVE-3813 URL: https://issues.apache.org/jira/browse/HIVE-3813 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Attachments: 0001-HIVE-3813-Allow-publishing-artifacts-to-an-arbitrary.patch, D7455.1.patch Allow publishing artifacts to an arbitrary remote repository by specifying -Dmvn.publish.repoUrl on the command line (patch by Thomas Dudziak). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3809) Concurrency issue in RCFile: multiple threads can use the same decompressor
Mikhail Bautin created HIVE-3809: Summary: Concurrency issue in RCFile: multiple threads can use the same decompressor Key: HIVE-3809 URL: https://issues.apache.org/jira/browse/HIVE-3809 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Priority: Critical RCFile is not thread-safe, even if each reader is only used by one thread as intended, because it is possible to return decompressors to the pool multiple times by calling close on the reader multiple times. Then, different threads can pick up the same decompressor twice from the pool, resulting in decompression failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3809) Concurrency issue in RCFile: multiple threads can use the same decompressor
[ https://issues.apache.org/jira/browse/HIVE-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3809: - Attachment: 0001-HIVE-3809-Decompressors-should-only-be-returned-to-t.patch Attaching a manually generated patch. Concurrency issue in RCFile: multiple threads can use the same decompressor --- Key: HIVE-3809 URL: https://issues.apache.org/jira/browse/HIVE-3809 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Priority: Critical Attachments: 0001-HIVE-3809-Decompressors-should-only-be-returned-to-t.patch, D7419.1.patch RCFile is not thread-safe, even if each reader is only used by one thread as intended, because it is possible to return decompressors to the pool multiple times by calling close on the reader multiple times. Then, different threads can pick up the same decompressor twice from the pool, resulting in decompression failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526840#comment-13526840 ] Mikhail Bautin commented on HIVE-3384: -- [~cdrome]: I think this patch has already been committed to 0.9 (thanks [~ashutoshc]!), and we need to deal with remaining test failures separately. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Assignee: Mikhail Bautin Priority: Minor Fix For: 0.11 Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, D6873.7.patch, HIVE-3384-2012-12-02.patch, HIVE-3384-2012-12-04.patch, HIVE-3384.2.patch, HIVE-3384-branch-0.9.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3384: - Attachment: HIVE-3384-2012-12-04.patch Manually generated patch corresponding to https://reviews.facebook.net/D6873?vs=onid=22815whitespace=ignore-all#differential-review-toc addressing the problem with PluginTest running from the pdk module. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, D6873.7.patch, HIVE-3384-2012-12-02.patch, HIVE-3384-2012-12-04.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[
https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509587#comment-13509587
]
Mikhail Bautin commented on HIVE-3384:
--
[~ashutoshc]: Thank you for testing the patch. I have addressed your comment
and PluginTest now passes when running from both {{builtin}} and {{pdk}}
modules.
{code:title=build/pdk/test-plugin/build/TEST-org.apache.hive.pdk.PluginTest.txt}
Testsuite: org.apache.hive.pdk.PluginTest
Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 42.592 sec
- Standard Error -
GLOBAL SETUP: Copying file:
file:/wd/hive-trunk/build/pdk/test-plugin/test/onerow.txt
Deleted file:/wd/hive-trunk/build/pdk/test-plugin/build/warehouse/onerow
org.apache.hive.pdktest.Rot13 SETUP: Deleted
file:/wd/hive-trunk/build/pdk/test-plugin/build/warehouse/rot13_data
org.apache.hive.pdktest.Rot13 TEARDOWN:
Hive history file=/tmp/mbautin/hive_job_log_mbautin_201212040011_1557635606.txt
OK
Time taken: 6.16 seconds
GLOBAL TEARDOWN:
Hive history file=/tmp/mbautin/hive_job_log_mbautin_201212040012_1607147323.txt
OK
Time taken: 0.287 seconds
- ---
Testcase: SELECT tp_rot13('Mixed Up!') FROM onerow; took 2.532 sec
Testcase: SELECT tp_rot13(s) FROM rot13_data; took 2.495 sec
{code}
HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC
specification
--
Key: HIVE-3384
URL: https://issues.apache.org/jira/browse/HIVE-3384
Project: Hive
Issue Type: Bug
Components: JDBC
Reporter: Weidong Bian
Priority: Minor
Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch,
D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, D6873.7.patch,
HIVE-3384-2012-12-02.patch, HIVE-3384-2012-12-04.patch, HIVE-3384.2.patch,
HIVE-3384.patch, HIVE-JDK7-JDBC.patch
jdbc module couldn't be compiled with jdk7 as it adds some abstract method in
the JDBC specification
some error info:
error: HiveCallableStatement is not abstract and does not override abstract
method TgetObject(String,ClassT) in CallableStatement
.
.
.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3384: - Attachment: HIVE-3384-branch-0.9.patch Attaching a version of the patch for the 0.9 branch. [~ashutoshc]: Thank you for taking care of this JIRA. Could you please commit this one to the 0.9 branch as well? HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Assignee: Mikhail Bautin Priority: Minor Fix For: 0.11 Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, D6873.7.patch, HIVE-3384-2012-12-02.patch, HIVE-3384-2012-12-04.patch, HIVE-3384.2.patch, HIVE-3384-branch-0.9.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin)
Mikhail Bautin created HIVE-3772: Summary: Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin) Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin)
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3772: - Attachment: HIVE-3772-2012-12-04.patch Attaching trunk patch. Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin) - Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Reporter: Mikhail Bautin Attachments: D7155.1.patch, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3384: - Attachment: HIVE-3384-2012-12-02.patch Uploading a patch corresponding to https://reviews.facebook.net/D6873?vs=onid=22815whitespace=ignore-all#differential-review-toc but generated manually so that it applies cleanly using the patch command. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, HIVE-3384-2012-12-02.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[
https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508518#comment-13508518
]
Mikhail Bautin commented on HIVE-3384:
--
[~ashutoshc]: PluginTest succeeds for me with my most recent patch.
{quote}
Testsuite: org.apache.hive.pdk.PluginTest
Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 33.596 sec
- Standard Error -
GLOBAL SETUP: Copying file: file:/wd/hive-trunk/builtins/test/onerow.txt
Deleted file:/wd/hive-trunk/build/builtins/warehouse/onerow
Copying file: file:/wd/hive-trunk/builtins/test/iris.txt
Deleted file:/wd/hive-trunk/build/builtins/warehouse/iris
org.apache.hive.builtins.UDAFUnionMap TEARDOWN:
Hive history file=/tmp/mbautin/hive_job_log_mbautin_201212022240_971726350.txt
GLOBAL TEARDOWN:
Hive history file=/tmp/mbautin/hive_job_log_mbautin_201212022240_805568604.txt
OK
Time taken: 8.473 seconds
OK
Time taken: 0.531 seconds
- ---
Testcase: SELECT size(UNION_MAP(MAP(sepal_width, sepal_length))) FROM iris took
3.454 sec
{quote}
HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC
specification
--
Key: HIVE-3384
URL: https://issues.apache.org/jira/browse/HIVE-3384
Project: Hive
Issue Type: Bug
Components: JDBC
Reporter: Weidong Bian
Priority: Minor
Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch,
D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch,
HIVE-3384-2012-12-02.patch, HIVE-3384.2.patch, HIVE-3384.patch,
HIVE-JDK7-JDBC.patch
jdbc module couldn't be compiled with jdk7 as it adds some abstract method in
the JDBC specification
some error info:
error: HiveCallableStatement is not abstract and does not override abstract
method TgetObject(String,ClassT) in CallableStatement
.
.
.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3748) QTestUtil should correctly find data files when running in the build directory
[
https://issues.apache.org/jira/browse/HIVE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Bautin resolved HIVE-3748.
--
Resolution: Not A Problem
Release Note: I found a way to achieve the same result without changing the
code. In fact, Hive does run unit tests from a directory one level above the
source root (e.g. ${hive.root}/ql, etc.) but data/conf/hive-site.xml specifies
the property to look for test data files one directory above.
QTestUtil should correctly find data files when running in the build directory
--
Key: HIVE-3748
URL: https://issues.apache.org/jira/browse/HIVE-3748
Project: Hive
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
Attachments: D7005.1.patch
Some parts of the the TestCliDriver test suite (i.e. some jar lookups)
require that the current directory is set to the build directory. This makes
QTestUtil correctly find data files when running either in the Hive source
root or in the build directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3748) QTestUtil should correctly find data files when running in the build directory
Mikhail Bautin created HIVE-3748: Summary: QTestUtil should correctly find data files when running in the build directory Key: HIVE-3748 URL: https://issues.apache.org/jira/browse/HIVE-3748 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Some parts of the the TestCliDriver test suite (i.e. some jar lookups) require that the current directory is set to the build directory. This makes QTestUtil correctly find data files when running either in the Hive source root or in the build directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3384: - Status: Patch Available (was: Open) Submitting patch for Jenkins testing. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873.1.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3731) Ant target to create a Debian package
Mikhail Bautin created HIVE-3731: Summary: Ant target to create a Debian package Key: HIVE-3731 URL: https://issues.apache.org/jira/browse/HIVE-3731 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor We need an Ant target to generate a Debian package with Hive binary distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3731) Ant target to create a Debian package
[ https://issues.apache.org/jira/browse/HIVE-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HIVE-3731: - Status: Patch Available (was: Open) Ant target to create a Debian package - Key: HIVE-3731 URL: https://issues.apache.org/jira/browse/HIVE-3731 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D6879.1.patch We need an Ant target to generate a Debian package with Hive binary distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502582#comment-13502582 ] Mikhail Bautin commented on HIVE-3384: -- [~cdrome]: This type of error sometimes happens when loading class files compiled with Java 6 into a Java 7 JVM. The -XX:-UseSplitVerifier JVM option helps even though some people say it is unsafe. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3694) Generate test jars and publish them to Maven
[ https://issues.apache.org/jira/browse/HIVE-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501586#comment-13501586 ] Mikhail Bautin commented on HIVE-3694: -- Added a patch for review (preliminary version) at https://reviews.facebook.net/D6843 Generate test jars and publish them to Maven Key: HIVE-3694 URL: https://issues.apache.org/jira/browse/HIVE-3694 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Mikhail Bautin Priority: Minor Attachments: D6843.1.patch It should be possible to generate Hive test jars and publish them to Maven so that other projects that rely on Hive or extend it could reuse its test library. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501747#comment-13501747 ] Mikhail Bautin commented on HIVE-3384: -- [~shane.huang], [~cdrome]: thanks a lot for working on this! I am interested in this patch being committed soon because it is annoying to keep applying these changes locally just to build Hive with JDK 7. [~cdrome]: to help move this patch forward, I have combined the two patches in and posted them on Phabricator on https://reviews.facebook.net/D6873 (trunk patch). The patch should show up on this JIRA within the next hour, but if it does not, I will upload it manually. I think it is better to go through a review loop on the trunk patch first and then do a backport for branch-0.9. I am happy to do the backport once the trunk patch is committed. HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873.1.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3694) Generate test jars and publish them to Maven
Mikhail Bautin created HIVE-3694: Summary: Generate test jars and publish them to Maven Key: HIVE-3694 URL: https://issues.apache.org/jira/browse/HIVE-3694 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Mikhail Bautin Priority: Minor It should be possible to generate Hive test jars and publish them to Maven so that other projects that rely on Hive or extend it could reuse its test library. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
