LZO Compression

2019-01-29 Thread dam6923
Hello, Quick question about LZO compression. After reading the docs, it seems to me that I must use DeprecatedLzoTextInputFormat in order to work with LZO files. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO However, I am not sure that is correct. As I understand it, as

Re: insert data into hadoop / hive cluster

2018-12-27 Thread dam6923
Check out an ETL tool such as StreamSets, NiFi, Pentaho. On Wed, Dec 26, 2018, 11:55 PM Daniel Takacs I'm working on an ETL that requires me to import a continuous stream of > CSVs into hadoop / hive cluster. For now let's assume the CSVs need to end > up in the same database.table. But the

Work Needs Review and Commit

2018-11-08 Thread dam6923
Hello Team, I've been trying to focus my latest work on making the logging of the service more concise and more useful. There's a lot of clutter in the logs that makes troubleshooting difficult. I've also submitted a few ideas for other small improvements. Can you please assist in reviewing

MapReduce Output File Names

2018-07-27 Thread dam6923
Hello, When Hive MapReduce jobs create HDFS output files, they use the format: 00_0.gz 00_0.gz_copy_1 00_0.gz_copy_2 00_0.gz_copy_3 ... This seems like it could become a long running list over time. In fact, the code says "leave the below loop for now until a better approach is

CREATE FUNCTION USING JAR

2018-06-09 Thread dam6923
Hello, In regards to using the "CREATE FUNCTION USING JAR" syntax, does the statement check for the following when the statement is submitted? 1. The existence of the JAR file 2. The JAR files contains the class specified in the statement? In a mulit-server environment, does this statement need

Global Configuration Read

2018-06-09 Thread dam6923
Hello, I am working on a patch for HiveServer2 whereby I will propose adding a new self-contained class which will have its own thread pool. I would like to make the size of the thread pool configurable from hive-site.xml at server start-up. All of the configurations I see are tied to the