[jira] [Created] (PARQUET-1237) Reading big texts cause OutOfMemmory Error. How to read text partialy?

[Andrei Iatsuk (JIRA)]

Andrei Iatsuk created PARQUET-1237:
--------------------------------------

             Summary: Reading big texts cause OutOfMemmory Error. How to read 
text partialy?
                 Key: PARQUET-1237
                 URL: https://issues.apache.org/jira/browse/PARQUET-1237
             Project: Parquet
          Issue Type: Bug
          Components: parquet-avro
    Affects Versions: 1.8.1
         Environment: I have dataset with big strings (every record about 15 
mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without 
reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader<GenericRecord> parquetReader = AvroParquetReader
    .<GenericRecord>builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read(); 
  // record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
            Reporter: Andrei Iatsuk


 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without 
reading all record?

 

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader<GenericRecord> parquetReader = AvroParquetReader

    .<GenericRecord>builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text 
in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)